Preventing CloudWatch Log Group Recreation During CloudFormation Stack Deletion

AWS CloudWatch Logs help in monitoring, storing, and accessing log files generated by various AWS services, like AWS Lambda, Amazon EC2. Log groups and log streams are the key resources of AWS CloudWatch Logs. Log stream stores log events from the same source, whereas a log group is a collection of log streams that share common properties. I have extensively used AWS CloudWatch Logs to store logs from Lambda functions. Log groups can be created in two ways: indirectly, when a Lambda function automatically creates one upon its first log event, or directly, by defining it explicitly within an AWS CloudFormation template.

AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation template to create a Lambda function

Resources:
  # Lambda Function
  MyServerlessFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: MyServerlessFunction
      Description: A sample Lambda function
      Handler: index.handler
      ....

Key points about the above stack:

1. The lambda function sends logs to a default log group named /aws/lambda/<function_name>.First invocation of the lambda function will trigger the log group’s creation. If the log group already exists, the function continues to create new log streams within it.
2. When the log group is not managed as part of the CloudFormation stack, deleting the stack deletes the Lambda function, leaving the log group behind. Such log groups are referred to as orphaned log groups because their generating resource no longer exists, yet they remain active in the system.
3. The cost associated with a log group depends on the log group’s retention period, and by default, CloudWatch stores log groups indefinitely. These orphaned log groups, if neglected and not deleted, can cost a lot.

AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation template to create a Lambda function, a CloudWatch Log Group and a custom resource

Resources: 
  # CloudWatch Log Group
  LambdaLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/MyServerlessFunction
      RetentionInDays: 14

  # Lambda Function
  MyServerlessFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: MyServerlessFunction
      Description: A sample Lambda function
      Handler: index.handler
      LoggingConfig:
        LogGroup: !Ref LambdaLogGroup
      ....
  
  # Custom resource
  MyCustomResource:
    Type: "Custom::TestResource"
    Properties: 
      ServiceToken: !GetAtt MyServerlessFunction.Arn

Key points about the above stack:

Get Ayushi Sharma’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in
1. A log group is explicitly attached to a Lambda by creating the log group as part of the stack. To use a different log group name in LoggingConfig, you can enter an existing log group or enter a new log group name.
2. You need to ensure there is no log group present with the same name as the LambdaLogGroup resource; otherwise, it will fail with resource already exists error.
3. !Ref creates an internal dependency(like DependsOn) between the lambda function and the log group. The lambda function uses a value of LambdaLogGroup, so the function is created only after the log group is created.
4. A custom resource calls the lambda function, and its logs are written to the function’s log group for all types of stack requests(Create, Update, and Delete).

A common issue arises when deleting a CloudFormation stack that contains both a Lambda function and a custom resource invoking that function. During stack deletion, CloudFormation processes delete actions aggressively and does not always respect the DependsOn attribute, which can complicate these types of deployment scenarios. Additionally, Lambda logging to AWS CloudWatch Logs is not real-time; it relies on an internal queue-based system that can delay log delivery(asynchronous log writes). Due to this delay, a Lambda function may attempt to write logs after its associated log group has already been deleted, causing CloudWatch Logs to recreate the log group automatically. This behavior can result in orphaned log groups that remain in the account and incur unnecessary costs. A hub-and-spoke deployment model is often preferred for Lambda functions and custom resources because it provides better separation of concerns and simplifies stack management.

The recreated log group will have a retention of Never Expire, which means logs will be kept indefinitely until you delete the log group or set a different retention period.

Lambda’s log group — Last event time of the log stream is 05:52:52 (UTC)

The Lambda function was deleted at 05:52:57 (UTC), followed by the deletion of its associated log group at 05:52:58 (UTC). Both timestamps occur after the last recorded event time in the log stream, indicating that all log events were successfully delivered and written before the log group deletion process began.

CloudTrail events indicate that the log group and log stream were recreated at 05:53:00 (UTC) by the Lambda function. This behavior reflects a delay in log event delivery. The pending log events were still in transit when the original log group was deleted, and upon their eventual delivery, CloudWatch automatically recreated the log group to store these late-arriving logs.

After reviewing multiple discussions on AWS re:Post, GitHub issues, and consulting with an AWS Lambda Support Engineer, I found a reliable solution to prevent the log group recreation issue. You can add a DENY permission to the Lambda execution role to block the creation of new log groups. Another possible workaround is to introduce an artificial delay between the deletion of the Lambda function and its log group. This allows sufficient time for any pending log events to be delivered. However, determining the appropriate delay duration can be challenging, as it depends on the number of log events; larger volumes may require a longer delay, while smaller ones may need only a brief pause.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Deny",
                "Action": [
                    "logs:CreateLogGroup"
                ],
                "Resource": "arn:aws:logs:<region>:<account_id>:log-group:/aws/lambda/<your_function_name>"
            }
        ]
    }