Can AI Kill Humans?
AI Won’t Kill Us. Unless We Choose to Build It Unsafely.
The past few weeks have been quite something on the AI front. We have had the OpenAI-Hugging Face fiasco (https://huggingface.co/blog/agent-intrusion-technical-timeline) and an Anthropic researcher quitting while citing fears about being involved in developing AI agents that could potentially kill humans within the next decade.
Unsurprisingly, such news makes people who are outside the AI world, people who don’t necessarily know how the technology works, wonder whether they are at risk, or whether their very survival is under threat. Last day, one of the non-AI guys asked me, after I told him I worked in AI:
“Can AI kill humans?”
I did have an answer to his question. And it is an answer that deserves a little more thought than a one-line response.
Yes, AI can kill. More precisely, mediocre AI design can kill humans and cause major societal harm. At the very least, I think we need to be honest about what we are actually asking. Instead of asking, “Can AI kill humans?”, we should ask a more uncomfortable question:
“Can the designers and deployers of autonomous AI systems cause harm because they have failed to do their jobs well?”
The answer to that is unequivocally yes.
When I was 15, my friend said he thought I could be a good doctor. With my level of impatience and restlessness, I didn’t think I’d make a particularly good doctor. Being a bad doctor, I explained, is a high-stakes job. Your bad day could involve a dead patient. I said I would choose a profession where the consequences of doing a bad job were less severe. I thought ‘hmm..maybe computers’. A bad program would just crash your PC. It wouldn’t take anyone’s life. Well, that is no longer necessarily true.
Traditional software generally does what its programmers tell it to do. An autonomous AI system is different. You might give it a goal such as: “Improve our cybersecurity.” And instead of specifying every action it should take, you give the system some degree of autonomy to determine how to achieve that goal. That changes the risk profile fundamentally.
So what does poor design look like when we build autonomous agents? It is, quite simply, designing an agent without being able to establish meaningful confidence that it will stay within its authorized boundaries and avoid harmful actions. We might assume that systems with this much agency must undergo thorough evaluation before deployment. But, how are they actually tested. In most cases, we merely observe that the agent ‘usually’ behaves safely under a set of test conditions. That is, we are measuring patterns of behavior rather than demonstrating that the agent’s architecture actually constrains what it can do.
This makes harmful agent behavior entirely plausible, and it has led to a serious question that has been debated for some time now: who is responsible when an autonomous agent makes a harmful decision? This is where things get complicated. Or perhaps, this is where we have allowed the complication to become a convenient feature of the system. The more difficult it is to establish who is responsible for an AI agent’s actions, the easier it becomes for the stakeholders building and deploying these systems to distance themselves from the consequences. And that creates a dangerous incentive: if accountability is sufficiently blurry, companies can argue that no single party could have reasonably anticipated or controlled what the agent did. That ambiguity can, in effect, become a license to ship half-cooked AI agents and products, powerful enough to take consequential actions, but not mature enough to reliably stay within their intended boundaries. European Union is at least trying to take steps toward regulating AI, while most of the other countries are engaged in a broader race for technological and economic dominance. The danger is that the race itself creates pressure to release systems before they are sufficiently mature. The incentives are obvious. Launch faster. Demonstrate more capability. Capture market share. Get investment. Increase valuation.
When does AI cause harm? Is it harmful only when it takes someone’s life or hacks into another network or does something legally unacceptable?
Get Savitha Sam Abraham’s stories in your inbox
Join Medium for free to get updates from this writer.
We are all familiar with the use of AI to shortlist CVs received for a job application. But an implicit requirement has now emerged: write your CV in such a way that the agent classifies it as a “Recommended CV”. We now have AI agents designed for this new application, agents that tell us how to write a CV so that a resume-filtering system cannot say “No.” And therein lies the problem. Instead of building a resume-filtering agent capable of identifying genuinely eligible candidates, we are making the task easier for the filtering agent by restricting and adjusting our own ability to express ourselves to the narrow space that the agent can interpret. Does this have consequences? It does not kill anyone. It merely emotionally drains a genuine candidate, erodes their confidence, but keeps them alive (mostly). Who is responsible for the harm caused? Is it the deployer of the agent or the one who designed it?
The EU AI Act places obligations on both providers and deployers of high-risk AI systems (CV screening is categorized as high-risk AI ). Although it sets out the goals, as of now, it does not fully specify the technical means of achieving them. For example, it requires high-risk AI systems, like CV-screening agents, to address risks such as bias, transparency and auditability. But there is no single technical specification that defines exactly what “unbiased” or “auditable” means in every context. This gap makes it difficult for companies to know exactly what they need to demonstrate to establish compliance, and equally difficult for regulators to determine consistently whether a system meets the required standard.
Consider a hypothetical (maybe not so hypothetical) situation.
Company A buys an autonomous AI agent from Company B. The agent is advertised as being capable of finding cybersecurity vulnerabilities and fixing them. When A deploys the agent, it discovers a way into a third-party network belonging to Company C and autonomously exploits it.
Did Company A instruct the agent to attack Company C? No, so should it be held accountable?
Did Company A take reasonable precautions to prevent the agent from doing exactly this? Did it give the agent excessive permissions, ignored previous incidents and didn’t sandbox it. Basically, was Company A negligent? Negligence could create liability depending on the law.
But, what if Company B built an agent whose safety mechanisms were inadequate or its developers failed to anticipate an obvious failure mode? Would it be fair to just hold the deployers liable? If society makes the deployer responsible for an autonomous agent’s actions, it must simultaneously force the seller to stand behind the reliability of the system they are selling. Otherwise, liability becomes a shield for mediocre AI design. The more autonomy a company sells, the greater the burden it should bear to demonstrate that the system can remain within its intended boundaries.
Here is a proposition worth considering: What if we start by holding the deployer accountable? What if the organization that chooses to deploy an autonomous AI agent is held accountable, for the consequences of deploying it? Company A chose to use an autonomous agent irrespective of the uncertainties existing around it. Even if Company A did not know exactly what the agent would do in every possible situation, it knew that autonomous systems carry risks. If a company is willing to take those risks, perhaps it should also be willing to bear some responsibility when those risks materialize. And this kind of accountability could create an interesting technological incentive. Imagine you are a company buying an autonomous agent. You would then care to ask:
“I need to know what this agent is going to do.”
But more importantly, you would want to know what it cannot do. In other words, you would be demanding safety guarantees: clear boundaries around the agent’s behavior and, crucially, actions that the agent is guaranteed not to take, regardless of the scenario it encounters. This could create a contractual relationship between the buyer and the seller. The seller would not simply be selling an agent that is expected to behave safely; they would be making explicit guarantees about the boundaries of its behavior. If those guarantees were violated, the buyer could invoke the remedies specified in the contract, potentially a legal action against the seller.
That is where liability becomes more than a legal question. It becomes an engineering incentive. It would make Reliability a Competitive Advantage. Today, AI companies have enormous incentives to compete on capability. “Look what our agent can do! It can write code, generate videos, operate your business, conduct research, create 3D animations and perform thousands of other tasks!” Those capabilities are exciting. They make great demos. They attract investors. They are the lifeline keeping the AI bubble from bursting. But there is another kind of achievement that is much harder to demonstrate in a product launch:
“We spent three years making sure the agent doesn’t accidentally destroy your database, spend all your money or violate somebody else’s system.”
Instead of incentivising an agent that can do 500,000 things (the curse of AGI!) but whose behavior becomes unpredictable when it encounters a new, never-before-observed scenario, let us celebrate an agent that can do 100 things extremely well, within clearly defined boundaries. Let the race be towards reliable AI architectures. The current data-driven approach of statistical pattern matching will always leave some scenarios unobserved. The world we live in is so astonishingly complex that it is impossible to anticipate every permutation and combination of events. There will always be that one scenario we missed. And I hope, there always will be. Because that unpredictability is part of what makes the real world so beautiful.
It is high time we think differently. It is like someone telling you there is gold at a particular location. You find a few gold coins, and because you have already invested in the most expensive mining equipment, you spend the next ten years digging deeper and deeper, convinced that a rich gold lode must be somewhere beneath you. But maybe the problem is not that you have not dug deep enough. It is that you are digging in the wrong place. Sometimes, digging a hole a few miles away may take you to diamonds. AI needs precisely that change in direction and I hope that one day we can all say, with confidence:
“The AI I use or the AI I built doesn’t kill or harm anyone.”
