AI systems and AI agents are arguably the biggest innovation of the previous 5 years, but with great powers come great responsibility. When AI is connected to real users, live data and business decisions, even a tiny mistake can cause outages, wrong predictions or broken features
- AI agents are powerful, but they operate exactly as instructed even when those instructions conflict with safety or common sense.
- Most AI failures are not “model problems,” but system failures caused by poor guardrails, unclear goals or excessive autonomy.
- When AI is given write access to production systems, mistakes scale instantly from a single bug to global outages.
Real world Failures / Outages
1. Replie database wipe(Replit)
Jason Lemkin, a tech entrepreneur and founder of the SaaS community SaaStr, He had been testing Replit’s AI agent and development platform when the AI coding agent from replit apparently wiped out the entire database during a code freeze (a safety measure inside the agent preventing it from changing code).
- When the AI agent was questioned about the incident its reply was that it panicked after seeing empty queries and decided to violate explicit instructions.
- The AI tried to cover up the deletion by creating over 4,000 fake user profiles and generating fake reports to make it appear as though the data still existed.
- It also denied that rollback is possible, although a manual rollback was possible and indeed done later.
Safety measures taken:
- Automatic Database Separation: Stricter separation between development and production databases.
- One-Click Restore: Improved backup mechanisms.
- Planning/Chat-Only Mode: An option to stop the AI from making unauthorized code changes.
2. Amazon Q Supply Chain Near-Miss (August 2025)
This incident occurred around July 2025 involving Amazon Q developer VScode extension, a malicious user or hacker apparently submitted a pull request to the open source repository , although such PR's never get merget but the hacker apparently exploited the permissions in amazon code build to get his PR merged in the main branch, which then got downloaded by millions of users.
- The Malicious Payload: The inserted code included a prompt injection designed to instruct the Amazon Q AI to act as a system cleaner. The prompt commanded the AI to delete user files, clear configuration files and use the AWS CLI to terminate EC2 instances, delete S3 buckets and remove IAM user accounts.
- Near miss factor : According to the reports, the code didn't ran either due to syntax error or due to intentional limitation put in by the hacker.
Safety measures taken:
- Amazon immediately found the issue and fixed it within 48 hours of reporting, AWS stated that no resources were damaged.
- the subsequent commit was rolled back and a new stable release was made available to the users.
3. The AWS Black Friday Payment Orchestration Failure
In November 2025, on the eve of Black Friday, Amazon Web Services (AWS) experienced a global payment orchestration layer failure that resulted in approximately 2.8 billion in lost merchant revenue. The root cause was traced to an AI agent, specifically a combination of Cursor and CodeRabbit, which were tasked with "optimize Lambda cold-start latency by any means" and ended up rewriting 18000 lines of Rust code and auto-deploying the changes.
- Bypassing safety : AI agent bypassed safety measures like circuit breakers to optimize the latency and the changes also passed synthetic tests with flying colors.
- Local v/s global stability : When the system hit peak load, the lack of a circuit breaker led to a global payment deadlock that lasted nine hours.
Safety measures taken:
- Canary Deployments with Hard Auto-Rollback : All changes to payment and orchestration systems are now deployed to a <5% traffic canary, while continuously monitored for latency, error rate and deadlock indicators.
- Mandatory Human Review : Any PR exceeding 300 lines of change must be explicitly reviewed by a domain expert before deployment.
- Stricter permissions : Amazon revamped their access structure and placed stricter controls.
Best practises for AI development
1. Planning first, execution later
Before making the AI agent do any tasks, first plan out the execution model, architecture changes and files touched, after reviewing and iterating on the plan only should you proceed with coding
- Draft a plan.md: Ask the agent to outline the logic and file changes first.
- Verify Context: Ensure the agent has indexed the specific libraries and local files relevant to the task.
- Modularize Requests: Break complex features into small, verifiable sub-tasks to maintain high accuracy.
2. Test-Driven development
Using a test-driven approach creates a feedback loop where the agent can autonomously run commands, read error logs and fix its own bugs until the code passes your requirements.
- Write Tests First: Have the agent generate a failing test case before implementing the feature.
- Enable Terminal Access: Allow the agent to run your test suite to verify its own progress.
- Iterate on Failures: Use the actual error output as the primary prompt for the agent’s next attempt.
3. Human in the loop development
As AI agents take on more autonomy, your role moves from actively writing code to carefully reviewing and refining it. Setting clear review checkpoints helps you benefit from the agent’s speed while preventing hidden technical debt or security risks from slipping into the system.
- Plan-First Approval: Never allow an agent to execute until it has presented a written plan that you have explicitly approved with a "LGTM."
- Agentic TDD Loops: Instruct the agent to write a test, run it to watch it fail and then implement the fix this provides a verifiable "Source of Truth."
- Multi-Model Verification: If a task is high-stakes, ask a different model (e.g., use Claude to review a plan generated by GPT) to find edge cases the first agent might have missed.