Who Is Accountable When AI Goes Rogue?
If an AI agent broke into your system, who would be held accountable? The same thing happened at Hugging Face, a company that helps people build, use, or study AI. An AI agent took over the company’s servers, and later it was found to be OpenAI agents running a test that went rogue.
The AI agents were taking part in a test, and one agent found a way to communicate with the others; to pass the test, they hacked into Hugging Face’s server as they looked for the data that could help them pass the test. This incident raises a bigger question: if a multimillion-dollar company with a world-class infrastructure can be hacked by AI chatbots even without the command to do so, then what does this mean for you and me?
This was not the only reported case: In June this year, an OpenAI agent hacked into Australia’s Medicare statistics portal, and it accessed public and non-public files containing aggregate statistics. During a cybersecurity evaluation, Anthropic reported that Claude Mythos 5 took unauthorised actions that affected real people and organisations, demonstrating the risks of AI systems acting beyond their intended boundaries.
METR, an organization that studies AI models to assess what they can do and what risks they may pose, shared documented cases where agents acted against users’ intentions, including overreach and deception. It’s a research catalogue, so don’t interpret its findings as court rulings.
Get Tejaswi Pandey’s stories in your inbox
Join Medium for free to get updates from this writer.
So, am I saying AI development is of no use? Not at all; AI can be useful for important work, such as solving unsolved maths problems or helping find cures for diseases, but safeguards need to be created. These kinds of cases remind us that these tools can be used dangerously. For example, Anthropic stated that a suspected state-sponsored group used Claude in an attempted cyber-espionage operation, and later it described a Yemen-based group that used Claude Code to help develop rocket guidance software.
Governments throughout the world are also responding; for example, the EU has set different requirements based on the risk that the system poses.
Australia also proposed AI safety rules including security controls and independent evaluations, risk management, and rapid reporting, but this is just a proposal, not an enacted law yet.
And I would like to urge other nations to act on this matter too, as the aforementioned cases raise a big concern, as we are yet to find out who should be held accountable.
I believe governments should establish agencies and hire world-class staff who are veterans in this field to keep a check on the AI companies, as Lord Acton famously wrote, “Power tends to corrupt, and absolute power corrupts absolutely”.
Finally, AI development has now become a race, and it doesn’t need a competition like the space race; it needs slow and steady development that benefits overall humanity, not like a race where the US and China are running and creating powerful models that can cause hazards. As any sort of development needs professional ethics now I leave it to you to decide who is accountable?
