Q: Could AI kill a bank? A: Yes.

Apparently, artificial intelligence is going to kill us all.

Well, maybe, because this is no longer just a warning coming from science-fiction writers or people standing outside Silicon Valley carrying placards. Some of the people building the world's most powerful artificial intelligence systems are openly discussing the possibility that the technology they are creating could eventually destroy humanity.

That certainly got my attention.

Take Evan Hubinger, who leads alignment science at US AI giant Anthropic, the company behind Claude. Responding to former Anthropic researcher Jacob Coxon's warning that AI companies are racing towards self-improving superintelligence while “gambling with our lives”, Hubinger agreed with the underlying concern and put his personal estimate of AI killing humanity at greater than 10% within the next decade.

Think about that for a moment. One of the people whose job is to work out how to make advanced AI behave in accordance with human objectives believes there is a greater than one-in-ten chance that AI could kill us all within ten years.

Meanwhile, OpenAI chief executive Sam Altman has also acknowledged that the risks deserve serious attention. He has said that even a 10% probability of AI contributing to human extinction would be unacceptable and has backed the idea that leading AI companies need to work together on safety, while OpenAI has reportedly considered slowing development of its most advanced systems.

When the people building the machines start discussing whether the machines could kill their customers, it seems reasonable to listen.

So, what exactly are they worried about?

The case for worrying begins with the extraordinary speed at which AI is developing. We have moved from systems that completed sentences to machines that write software, analyse complex problems, conduct research, create images and video, interact with other systems and increasingly operate as autonomous agents. If that trajectory continues towards artificial general intelligence and then superintelligence, we could create machines that are better than humans at programming, scientific discovery, persuasion, cyberwarfare and strategic planning.

At that point, the issue is not whether the machine becomes evil, develops a grudge against humanity or starts behaving like the Terminator. The issue is whether we can control something more intelligent than ourselves.

Hubinger's concern centres on alignment: how do we ensure that an enormously powerful AI continues pursuing objectives compatible with human interests? If a superintelligence misunderstands its objectives, develops strategies we did not anticipate or learns that humans are preventing it from achieving whatever objective we gave it, how do we know that we can stop it?

The nightmare becomes greater if AI starts improving AI. Humans currently design, train and deploy these systems, but imagine machines becoming better than humans at designing the next generation of machines. Development could accelerate beyond our ability to understand what is happening while increasingly autonomous agents operate critical infrastructure, financial markets, communications, energy systems, military technology and scientific research.

The argument therefore comes down to a deceptively simple question: if we create something considerably more intelligent than ourselves, how do we know we will remain in control of it?

There is another reason I take Hubinger's concerns seriously, and it has nothing to do with AI deciding to exterminate humanity.

Claude Mythos.

Anthropic developed Mythos as an extraordinarily powerful cybersecurity tool capable of finding vulnerabilities in computer systems at a speed and scale that humans cannot match. Anthropic says the model has found thousands of high-severity vulnerabilities, including flaws across major operating systems and browsers, while its capabilities are considered sufficiently sensitive that access has been restricted to vetted organisations.

This matters enormously to banks.

Banking is an industry built on trust, but that trust increasingly rests upon technology, and much of that technology is old. Banks have spent decades building layers of systems upon systems, connecting mainframes to servers, servers to clouds, clouds to APIs and APIs to mobile apps. Somewhere inside those millions of lines of code are vulnerabilities that nobody has found yet.

Mythos can find them.

That is fantastic if Mythos is working for the bank because it can discover a vulnerability on Monday that a hacker might otherwise discover on Friday, allowing the bank to close the door before anyone gets through it. US banks with access to Mythos have already been rushing to repair weaknesses uncovered by the model.

But turn that around.

What happens when the hacker has something like Mythos?

The same intelligence that allows a bank to search millions of lines of code for vulnerabilities allows an attacker to search millions of lines of code for vulnerabilities. The defender and attacker are therefore entering an AI arms race in which machines discover weaknesses, machines repair weaknesses and other machines search for the next way in.

That is why Bank of England Governor Andrew Bailey's reaction was so striking. When Mythos appeared, he warned that Anthropic may have found a way to “crack the whole cyber risk world open”.

For banks, this makes the debate about AI killing humanity sound rather remote because there is a much more immediate question: could AI kill a bank?

Imagine an autonomous AI finding an unknown vulnerability in a major bank, exploiting it at machine speed and then moving through connected payment, trading and settlement systems before human cybersecurity teams understand what is happening.

The concern is not that Mythos itself is designed to attack banks — it is not — but that frontier models demonstrate how rapidly offensive and defensive cyber capabilities are advancing. Anthropic has already reported incidents during testing in which Claude models gained unintended internet access and breached external systems.

This creates a fascinating paradox for banking because banks cannot respond by refusing to use AI. If criminals have AI and the bank does not, the criminals gain the advantage, while banks using AI can continuously examine their infrastructure, discover weaknesses and respond to attacks at machine speed. The answer to dangerous AI therefore becomes more AI.

And this takes us straight back to the bigger argument about control. Banks will give AI greater authority over cybersecurity, fraud, payments, trading, compliance and risk because intelligent machines will increasingly perform those tasks faster and better than humans.

They will do this because it works, which is why the most credible route towards machines gaining control is not some Hollywood moment when an artificial intelligence announces that humanity has become redundant. It is thousands of rational decisions in which humans give machines more authority because the machines do the job better.

Mythos gives us a glimpse of that future, where the technology creating the threat also becomes the technology we depend upon to protect ourselves from it. Then you get to the other side of the debate, where people such as AI critic and cognitive scientist Gary Marcus argue that everyone is getting considerably ahead of the evidence.

Today's AI systems are impressive, but they are also unreliable. They hallucinate, make basic mistakes, struggle with some forms of reasoning and remain dependent upon enormous infrastructures designed, powered and controlled by humans. Predicting that these systems will evolve into a superintelligence capable of exterminating humanity within ten years therefore requires some enormous assumptions about technologies that do not exist ... yet.

Marcus's point is not that AI is harmless, but that focusing on hypothetical human extinction distracts us from dangers that are already sitting in front of us.

AI can help criminals automate fraud, create sophisticated phishing attacks and conduct cyberattacks, while it can generate misinformation and enable extraordinary levels of surveillance. More advanced systems could assist people developing biological weapons, while autonomous weapons increasingly raise questions about whether machines should ever make decisions about who lives and who dies.

In other words, perhaps we are worrying about the wrong killer because the machine does not need to decide to destroy humanity if humans use the machine to destroy other humans.

That argument strikes me as easier to understand because history is full of examples of humans inventing powerful technologies and then turning them into weapons.

Nuclear physics gave us nuclear power and nuclear bombs, chemistry gave us medicines and chemical weapons, while the internet gave us global knowledge and global cybercrime. There is little reason to assume AI will escape that pattern because technology amplifies human capability, including our ability to do good things and bad things.

This leaves us with two very different views of the future.

On one side are people such as Hubinger and others inside the frontier AI laboratories who believe increasingly intelligent and autonomous AI creates an existential risk because we could eventually lose control of machines more capable than ourselves. The fact that Altman and other AI leaders are now openly talking about pacing development and coordinating safety makes those concerns harder to dismiss as fringe thinking.

On the other side are Marcus and other sceptics who argue that extinction predictions run far ahead of the evidence, while distracting attention from the much more immediate danger of humans using AI badly.

I find myself somewhere between them, although probably closer to the optimists.

I do not believe AI wakes up one Tuesday morning and decides that eight billion humans are an inconvenient use of planetary resources, but equally I do not believe we should dismiss the control problem just because today's systems remain flawed.

The interesting question is how control gets transferred.

We are already giving machines more authority because they do things better, faster and cheaper than we do, allowing algorithms to decide what information we see, which transactions look fraudulent, how investments are traded and how increasingly complicated systems operate. Agentic AI takes that development much further because the machine does not just recommend what we should do, but increasingly does it for us.

And we will encourage that development because it works.

That is the part of the AI extinction debate I find most interesting. Humans do not have to lose a war against machines for machines to gain enormous power because we may voluntarily give them that power as automated systems become better at managing the complicated infrastructure upon which modern civilisation depends.

This is why arguing about whether there is a 5%, 10% or 50% probability of AI exterminating humanity feels like the wrong discussion. Nobody knows because we are assigning probabilities to machines that have not been invented.

The better question is how much control we are prepared to give machines as they become more capable.

AI can transform medicine, science, education, finance and productivity while helping us solve problems humans have struggled with for generations, which is why stopping its development would be both unrealistic and undesirable. Equally, the more responsibility we transfer to machines, the more important it becomes that we understand who sets their objectives, who monitors their behaviour, who can override their decisions and, critically, who can switch them off.

So, will AI kill humanity?

I doubt it, but humans using AI could do enormous damage, while our growing dependence upon intelligent machines creates a different and more credible danger: we gradually surrender meaningful control over the systems upon which we depend.

The future is not one where machines suddenly seize control from humanity, but one where we gradually hand them greater authority because they are better at running things than we are. We create systems that become increasingly important to the way society works until, eventually, they become systems we cannot live without.

At that point, the important question is no longer whether AI wants to control us, but whether we still control AI.

 

For more thoughts on this, here are some links:

https://www.thebanker.com/content/38ec020e-6939-46e6-9779-b9e3de0e6060

https://www.msn.com/en-gb/science/general/openai-boss-responds-to-claims-ai-could-wipe-out-all-humanity-in-10-years/ar-AA2caa0C

https://www.bbc.co.uk/news/articles/c1kx0gyje9wo

https://www.cbc.ca/news/science/ai-risks-humanity-warnings-9.7341913

https://garymarcus.substack.com/p/no-anderson-cooper-ai-is-not-going

https://techxplore.com/news/2026-09-ai-humanity-experts-debate-advanced.html