Almost Went Bankrupt Because My Lambda Function Wouldn’t Stop Calling Itself

The Scenario

You accidentally deploy a Lambda function that recursively invokes itself. Because the function is configured for automatic scaling, invocations continue for 14 hours and generate a massive AWS bill. How would you detect what is happening, stop it, identify the root cause, and prevent this from happening again?

Imagine a company has an image-processing system:

User uploads image
       ↓
      S3
       ↓
    Lambda
       ↓
Resize / compress
       ↓
Store processed image in S3

Normally, everything is straightforward:

image.jpg uploaded
      ↓
Lambda runs
      ↓
image resized
      ↓
processed-image.jpg

But a developer accidentally configures Lambda to write the processed image back into the same S3 location that triggers the Lambda.

Now the system becomes:

S3 upload
   ↓
Lambda
   ↓
S3 upload
   ↓
Lambda
   ↓
S3 upload
   ↓
Lambda
   ↓
...

A tiny configuration mistake has created a feedback loop.

Trigger & Reason (why it happened)

Lambda’s output generated the event that invoked Lambda in the first place.

• The bucket has an event notification: “new file → invoke Lambda”
• Lambda’s own output is also a “new file” landing in that same bucket
• So every time Lambda finishes, it accidentally creates a brand new trigger event for itself
• This repeats endlessly — Lambda didn’t “decide” to loop, the architecture looped by accident

Find the Real Root Cause

I will not spend 30 minutes investigating the code while the loop is still generating events.

The first objective is:

Stop additional executions.

Step 1 — Set Lambda Reserved Concurrency to 0

Temporarily set the function’s reserved concurrency to zero.

Lambda
  ↓
Reserved concurrency = 0

This prevents new invocations from executing.

Step 2 — Disable the Trigger

Next, stop the event source generating the invocations.

For an S3-triggered Lambda, disable/remove the problematic event notification or otherwise stop the event path.

For another event source, such as SQS, you might disable the event source mapping.

Event source
     ↓
   STOP
     ↓
  Lambda

Why do both?

Because stopping Lambda alone doesn’t necessarily address events already being generated or queued by the source.

Once the system is stable, I would start investigating.

I’d trace the complete flow.

First:

What is invoking Lambda?

S3 → Lambda

Then:

What is Lambda doing after it runs?

Lambda → writes processed image to S3

Then I ask:

Get AyushK’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Where exactly is it writing?

And I discover that it’s writing to the same bucket/prefix that triggers the Lambda.

So now I can explain the actual root cause:

The Lambda output was written to the same S3 location being monitored by its own trigger, so the output generated another input event.

I would also check the recent deployment

Since the scenario says this happened because of an accidental deployment, I’d check what changed immediately before the incident.

For example:

10:00 AM → New Lambda version deployed
10:03 AM → Lambda invocations start increasing

Then I’d compare the old and new configuration/code.

Maybe somebody changed:

output_bucket = "images"

when it should have been:

output_bucket = "processed-images"

Or maybe an S3 prefix filter was removed.

That gives me a much better root-cause statement:

A recent deployment changed the Lambda output location or trigger configuration, causing Lambda’s output to become its own input.

How I would fix it

The cleanest solution is to separate the input and output.

Instead of:

Upload bucket
     ↓
   Lambda
     ↓
Same bucket

I’d use:

Upload bucket
     ↓
   Lambda
     ↓
Processed bucket

For example:

s3://image-upload
        ↓
      Lambda
        ↓
s3://processed-images

Now when Lambda creates the processed image, that object doesn’t trigger the image-processing Lambda.

Another option is using prefixes

If using two buckets isn’t necessary, I can also separate the objects using prefixes.

For example:

/images/input/
       ↓
     Lambda
       ↓
/images/processed/

The S3 trigger only watches:

/images/input/

Lambda writes to:

/images/processed/

So the processed image doesn’t match the trigger.

This is basically creating a boundary between input data and output data.

How I would prevent this in the future

I wouldn’t rely on a single protection. I’d use multiple layers:

1. Set concurrency limits
   Keep a reasonable limit so a bug can’t consume unlimited Lambda concurrency. This limits the blast radius, but doesn’t fix the bug.
2. Add CloudWatch alarms
   Monitor abnormal spikes in invocations, concurrency, errors, and throttles, and alert through SNS.
3. Monitor AWS costs
   Use cost monitoring/anomaly detection so an unexpected Lambda usage spike is caught quickly.
4. Make Lambda idempotent
   If the same event arrives twice, detect that it was already processed and skip it. This makes duplicate events safer.
5. Don’t rely only on recursive-loop protection
   AWS provides protection for some recursive patterns, but I’d still design the system properly with event filtering, monitoring, concurrency limits, and safe input/output separation.