<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Janngoncearuc on Medium]]></title>
        <description><![CDATA[Stories by Janngoncearuc on Medium]]></description>
        <link>https://medium.com/@janngoncearuc?source=rss-6c479913df22------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZnTJr52K5WyoNaL6</url>
            <title>Stories by Janngoncearuc on Medium</title>
            <link>https://medium.com/@janngoncearuc?source=rss-6c479913df22------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:52:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@janngoncearuc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Elastic Cloud Cluster with Terraform via Private Link endpoint on Google Cloud]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@janngoncearuc/elastic-cloud-cluster-with-terraform-via-private-link-endpoint-on-google-cloud-fb3bf7482676?source=rss-6c479913df22------2"><img src="https://cdn-images-1.medium.com/max/2600/1*bS249Dh0NrTjQI1RO2gQwg.png" width="3016"></a></p><p class="medium-feed-snippet">When it comes to deploying Elasticsearch securely in the cloud, ensuring private connectivity is crucial. Google Cloud offers Private&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@janngoncearuc/elastic-cloud-cluster-with-terraform-via-private-link-endpoint-on-google-cloud-fb3bf7482676?source=rss-6c479913df22------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@janngoncearuc/elastic-cloud-cluster-with-terraform-via-private-link-endpoint-on-google-cloud-fb3bf7482676?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb3bf7482676</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Thu, 20 Feb 2025 19:54:04 GMT</pubDate>
            <atom:updated>2025-02-20T19:54:04.753Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Is Ansible Still Used? ]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">If you&#x2019;ve been in the DevOps world for a while, you&#x2019;ve probably heard of Ansible&#x200A;&#x2014;&#x200A;the open-source automation tool that made configuration&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@janngoncearuc/is-ansible-still-used-c76b69d43a27?source=rss-6c479913df22------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@janngoncearuc/is-ansible-still-used-c76b69d43a27?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/c76b69d43a27</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Mon, 17 Feb 2025 20:27:12 GMT</pubDate>
            <atom:updated>2025-02-17T20:27:12.572Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Is AI Helping DevOps to Evolve?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">The DevOps Evolution&#x200A;&#x2014;&#x200A;Now With AI!</p><p class="medium-feed-link"><a href="https://medium.com/@janngoncearuc/is-ai-helping-devops-to-evolve-f9a3ed06cd26?source=rss-6c479913df22------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@janngoncearuc/is-ai-helping-devops-to-evolve-f9a3ed06cd26?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9a3ed06cd26</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Mon, 17 Feb 2025 20:22:24 GMT</pubDate>
            <atom:updated>2025-02-17T20:22:24.562Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Streaming S3 Logs to CloudWatch with AWS Lambda]]></title>
            <link>https://medium.com/@janngoncearuc/streaming-s3-logs-to-cloudwatch-with-aws-lambda-2470d5b00c7a?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/2470d5b00c7a</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Tue, 11 Feb 2025 06:01:38 GMT</pubDate>
            <atom:updated>2025-02-11T06:01:38.331Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Logging is a crucial part of any cloud-based infrastructure. When storing logs in Amazon S3, you may want an efficient way to ingest and analyse them in AWS CloudWatch. This allows for real-time monitoring, troubleshooting, and alerting on important events.</p><p>One way to achieve this is by creating an AWS Lambda function that automatically processes incoming S3 logs, creates the necessary CloudWatch log groups and log streams, and then publishes the logs.</p><p>In this article, we’ll walk through the process of setting up such a solution using AWS Lambda, S3, and CloudWatch. We’ll also include Terraform as an infrastructure-as-code approach to define our resources.</p><h3>Why Stream S3 Logs to CloudWatch?</h3><p>Storing logs in S3 is useful for archival and cost-effective storage. However, it lacks real-time monitoring and alerting capabilities. By streaming logs to CloudWatch, you can:</p><ul><li><strong>Gain real-time insights</strong> into system events.</li><li><strong>Set up alerts</strong> for critical errors or unusual patterns.</li><li><strong>Easily query logs</strong> using AWS CloudWatch Logs Insights.</li><li><strong>Centralize logs</strong> for better visibility across multiple AWS services.</li></ul><h3>Solution Overview</h3><p>The solution consists of:</p><ol><li><strong>An S3 bucket</strong> where logs are stored.</li><li><strong>A Lambda function</strong> that triggers on new log files in the bucket.</li><li><strong>A CloudWatch log group &amp; stream</strong> where processed logs are published.</li><li><strong>IAM permissions</strong> to allow Lambda access to S3 and CloudWatch.</li></ol><h3>AWS Lambda for Log Processing</h3><p>AWS Lambda will be configured to trigger whenever new log files are added to an S3 bucket. The function will:</p><ul><li>Read the incoming log files from folders.</li><li>Parse the log data.</li><li>Create a CloudWatch log group and stream if they don’t exist.</li><li>Publish the logs to CloudWatch.</li></ul><p>This enables efficient log ingestion and near real-time observability.</p><h3>Terraform Configuration</h3><p>To implement this solution in an automated and scalable way, we use Terraform. With Terraform, we define the S3 bucket, Lambda function, IAM roles, and CloudWatch log groups as code.</p><pre>resource &quot;aws_iam_role&quot; &quot;lambda_role&quot; {<br>  name = &quot;s3-to-cw-role&quot;<br>  assume_role_policy = templatefile(&quot;${path.module}/iam_policies/iam_role.json&quot;,<br>    {<br>    })<br>}<br><br># Attach policy to IAM Role<br>resource &quot;aws_iam_role_policy&quot; &quot;lambda_policy&quot; {<br>  name  = &quot;s3-to-cw-role-policy&quot;<br>  role  = aws_iam_role.lambda_role.id<br>  policy = templatefile(&quot;${path.module}/iam_policies/iam_policy.json&quot;,<br>    {<br>      aws_region         = &quot;AWS_REGION&quot;<br>      aws_account        = &quot;AWS_ACCOUNT&quot;<br>      ses_name           = &quot;SES_SENDER_NAME&quot;<br>      aws_bucket         = &quot;BUCKET_NAME_TO_CONVERT&quot;<br>      topic_arn          = &quot;TOPIC_ARN_FOR_ALERTING&quot;<br>    })<br>}<br><br>resource &quot;aws_lambda_function&quot; &quot;s3_to_cloudwatch&quot; {<br>  filename         = data.archive_file.lambda_zip.output_path<br>  function_name    = &quot;s3-log-processor&quot;<br>  role             = aws_iam_role.lambda_role.arn<br>  handler          = &quot;lambda_function.lambda_handler&quot;<br>  runtime          = &quot;python3.9&quot;<br>  source_code_hash = filebase64sha256(data.archive_file.lambda_zip.output_path)<br>  timeout          = 900<br>  memory_size      = 512<br>  environment {<br>    variables = {<br>      SENDER_EMAIL         = &quot;alerting@test.com&quot;<br>      REGION               = &quot;AWS_REGION&quot;<br>      LAMBDA_FUNCTION_NAME = &quot;arn:aws:lambda:AWS_REGION:AWS_ACCOUNT:function:s3-log-processor&quot;<br>      RECIPIENT_EMAIL      = &quot;SLACK_CHANNEL&quot;<br>    }<br>  }<br>}<br><br>data &quot;archive_file&quot; &quot;lambda_zip&quot; {<br>  type        = &quot;zip&quot;<br>  source_dir  = &quot;${path.module}/src&quot;<br>  output_path = &quot;${path.module}/lambda.zip&quot;}</pre><pre>resource &quot;aws_s3_bucket&quot; &quot;log_bucket&quot; {<br>  bucket = &quot;my-s3-log-bucket&quot;<br>}</pre><pre>resource &quot;aws_s3_bucket_notification&quot; &quot;log_notification&quot; {<br>  bucket = aws_s3_bucket.log_bucket.id</pre><pre>lambda_function {<br>    lambda_function_arn = aws_lambda_function.s3_to_cloudwatch.arn<br>    events              = [&quot;s3:ObjectCreated:*&quot;]<br>  }<br>}</pre><pre>resource &quot;aws_cloudwatch_log_group&quot; &quot;s3_log_group&quot; {<br>  name = &quot;s3-log-group&quot;<br>}</pre><pre>resource &quot;aws_lambda_permission&quot; &quot;allow_s3&quot; {<br>  statement_id  = &quot;AllowS3Invoke&quot;<br>  action        = &quot;lambda:InvokeFunction&quot;<br>  function_name = aws_lambda_function.s3_to_cloudwatch.function_name<br>  principal     = &quot;s3.amazonaws.com&quot;<br>  source_arn    = aws_s3_bucket.log_bucket.arn<br>}</pre><p>Here is are the policies for the role and policy itself:</p><p>A. Lambda role</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;Service&quot;: &quot;lambda.amazonaws.com&quot;<br>            },<br>            &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br>        }<br>    ]<br>}</pre><p>B. Lambda policy</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;ses:*&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:ses:${aws_region}:${aws_account}:identity/${ses_name}&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::${aws_bucket}*/*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogGroup&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;,<br>                &quot;logs:StartQuery&quot;,<br>                &quot;logs:GetQueryResults&quot;,<br>                &quot;logs:DescribeLogGroups&quot;,<br>                &quot;logs:DescribeLogStreams&quot;,<br>                &quot;logs:ListTagsLogGroup&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;sns:Subscribe&quot;,<br>                &quot;sns:Receive&quot;,<br>                &quot;sns:ListSubscriptions&quot;,<br>                &quot;sns:GetTopicAttributes&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;${topic_arn}&quot;<br>        }<br>    ]<br>}</pre><h3>AWS Lambda for Log Processing</h3><p>The Lambda function dynamically reads the bucket name from the event payload, ensuring flexibility for different S3 sources. It processes logs in batches, adhering to AWS CloudWatch’s size and count limits for log event uploads. This batching mechanism optimizes performance and prevents exceeding CloudWatch’s API constraints.</p><p>AWS Lambda will be configured to trigger whenever new log files are added to an S3 bucket. The function will:</p><ul><li>Read the incoming log file.</li><li>Parse the log data.</li><li>Create a CloudWatch log group and stream if they don’t exist.</li><li>Publish the logs to CloudWatch.</li><li>Send failure alerts via Amazon SES if any processing issues occur.</li></ul><pre>import boto3<br>import gzip<br>import json<br>from botocore.exceptions import ClientError<br>from urllib.parse import unquote_plus<br>from datetime import datetime<br>import re<br>import os<br><br>logs_client = boto3.client(&#39;logs&#39;)<br>s3_client = boto3.client(&#39;s3&#39;)<br>ses_client = boto3.client(&#39;ses&#39;)<br><br># Regular expression to match ISO 8601 timestamps<br>ISO8601_TIMESTAMP_PATTERN = re.compile(r&#39;\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z&#39;)<br><br>sender_email = os.environ[&#39;SENDER_EMAIL&#39;]<br>recipient_email = os.environ[&#39;RECIPIENT_EMAIL&#39;]<br>aws_region = os.environ[&#39;REGION&#39;]<br>lambda_function_name = os.environ[&#39;LAMBDA_FUNCTION_NAME&#39;]<br><br># Limits for CloudWatch Logs<br>MAX_BATCH_SIZE = 1048576  # 1 MB<br>MAX_BATCH_COUNT = 10000   # Maximum number of log events per batch<br>LOG_EVENT_OVERHEAD = 26   # Byte overhead per log event<br>def send_failure_alert(error_message, lambda_function_name, log_group_name, aws_region):<br>    &quot;&quot;&quot;Send email alert using Amazon SES for Lambda execution failure, with links to the Lambda function and CloudWatch Logs.&quot;&quot;&quot;<br>    subject = &quot;S3 to CloudWatch Lambda failed - Please action!&quot;<br><br>    lambda_console_link = f&quot;https://{aws_region}.console.aws.amazon.com/lambda/home?region={aws_region}#/functions/{lambda_function_name}?tab=configuration&quot;<br>    cloudwatch_console_link = f&quot;https://{aws_region}.console.aws.amazon.com/cloudwatch/home?region={aws_region}#logStream:group={log_group_name}&quot;<br><br>    body_html = f&quot;&quot;&quot;<br>    &lt;html&gt;<br>    &lt;head&gt;&lt;/head&gt;<br>    &lt;body style=&quot;font-family: Arial, sans-serif; color: #333;&quot;&gt;<br>      &lt;h2 style=&quot;color: #D9534F;&quot;&gt;Lambda Processing Failure Alert&lt;/h2&gt;<br>      &lt;p&gt;An error occurred in the &lt;strong&gt;{lambda_function_name}&lt;/strong&gt; Lambda function:&lt;/p&gt;<br>      &lt;table style=&quot;border-collapse: collapse; width: 100%;&quot;&gt;<br>        &lt;tr&gt;<br>          &lt;td style=&quot;border: 1px solid #ddd; padding: 8px; background-color: #f9f9f9;&quot;&gt;&lt;strong&gt;Error Message&lt;/strong&gt;&lt;/td&gt;<br>          &lt;td style=&quot;border: 1px solid #ddd; padding: 8px; background-color: #fefefe;&quot;&gt;{error_message}&lt;/td&gt;<br>        &lt;/tr&gt;<br>      &lt;/table&gt;<br>      &lt;br&gt;<br>      &lt;p&gt;Further details can be investigated via the following links:&lt;/p&gt;<br>      &lt;ul&gt;<br>        &lt;li&gt;&lt;a href=&quot;{lambda_console_link}&quot; style=&quot;text-decoration: none; color: #0073e6;&quot;&gt;Lambda Function Dashboard&lt;/a&gt;&lt;/li&gt;<br>        &lt;li&gt;&lt;a href=&quot;{cloudwatch_console_link}&quot; style=&quot;text-decoration: none; color: #0073e6;&quot;&gt;CloudWatch Log Group&lt;/a&gt;&lt;/li&gt;<br>      &lt;/ul&gt;<br>      &lt;p&gt;Ensure you are authenticated into the AWS Console to view the above links.&lt;/p&gt;<br>    &lt;/body&gt;<br>    &lt;/html&gt;<br>    &quot;&quot;&quot;<br><br>    try:<br>        ses_client.send_email(<br>            Source=sender_email,<br>            Destination={&#39;ToAddresses&#39;: [recipient_email]},<br>            Message={<br>                &#39;Subject&#39;: {&#39;Data&#39;: &#39;Lambda Processing Failure Alert&#39;},<br>                &#39;Body&#39;: {&#39;Html&#39;: {&#39;Data&#39;: body_html}}<br>            }<br>        )<br>    except Exception as e:<br>        print(f&quot;Error sending failure alert email: {str(e)}&quot;)<br><br>def lambda_handler(event, context):<br>    print(&quot;Lambda function started&quot;)<br><br>    try:<br>        for record in event.get(&#39;Records&#39;, []):<br>            if &#39;s3&#39; in record:<br>                process_s3_records([record], context)<br>            else:<br>              break<br><br>    except Exception as e:<br>        error_message = f&quot;Error processing logs: {str(e)}&quot;<br>        print(error_message)<br>        send_failure_alert(error_message, lambda_function_name, &quot;UnknownLogGroup&quot;, aws_region)<br><br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: json.dumps(&#39;Log processing complete!&#39;)<br>    }<br><br>def process_s3_records(s3_records, context):<br>    for s3_record in s3_records:<br>        if &#39;s3&#39; not in s3_record:<br>            continue<br><br>        bucket_name = s3_record[&#39;s3&#39;][&#39;bucket&#39;][&#39;name&#39;]<br>        key = unquote_plus(s3_record[&#39;s3&#39;][&#39;object&#39;][&#39;key&#39;])<br><br>        log_group_name = f&quot;/aws/s3/{bucket_name}&quot;<br>        log_stream_name = f&quot;{context.aws_request_id}_{datetime.utcnow().strftime(&#39;%Y%m%d%H%M%S&#39;)}&quot;<br><br>        ensure_log_group(log_group_name)<br>        sequence_token = ensure_log_stream(log_group_name, log_stream_name)<br><br>        try:<br>            response = s3_client.get_object(Bucket=bucket_name, Key=key)<br>            content = response[&#39;Body&#39;].read()<br>        except ClientError as e:<br>            error_message = f&quot;Error retrieving S3 object: {e}&quot;<br>            print(error_message)<br>            send_failure_alert(error_message, lambda_function_name, log_group_name, aws_region)<br>            continue<br><br>        if key.endswith(&quot;.log.gz&quot;):<br>            try:<br>                content = gzip.decompress(content)<br>            except OSError as e:<br>                error_message = f&quot;Error decompressing gzip file: {e}&quot;<br>                print(error_message)<br>                send_failure_alert(error_message, lambda_function_name, log_group_name, aws_region)<br>                continue<br><br>        log_entries = content.decode(&#39;utf-8&#39;).splitlines()<br>        print(f&quot;Processing {len(log_entries)} log entries from {key}&quot;)<br><br>        batches = create_batches(log_entries)<br>        send_batches(log_group_name, log_stream_name, batches, sequence_token)<br><br>def ensure_log_group(log_group_name):<br>    &quot;&quot;&quot;Ensure that the log group exists.&quot;&quot;&quot;<br>    try:<br>        response = logs_client.describe_log_groups(logGroupNamePrefix=log_group_name, limit=1)<br>        if not any(lg[&#39;logGroupName&#39;] == log_group_name for lg in response[&#39;logGroups&#39;]):<br>            logs_client.create_log_group(logGroupName=log_group_name)<br>    except ClientError as e:<br>        print(f&quot;Error ensuring log group {log_group_name}: {e}&quot;)<br><br>def ensure_log_stream(log_group_name, log_stream_name):<br>    &quot;&quot;&quot;Ensure that the log stream exists and return its upload sequence token if available.&quot;&quot;&quot;<br>    try:<br>        streams_response = logs_client.describe_log_streams(<br>            logGroupName=log_group_name,<br>            logStreamNamePrefix=log_stream_name,<br>            limit=1<br>        )<br>        if not streams_response[&#39;logStreams&#39;]:<br>            logs_client.create_log_stream(logGroupName=log_group_name, logStreamName=log_stream_name)<br>            return None<br>        else:<br>            return streams_response[&#39;logStreams&#39;][0].get(&#39;uploadSequenceToken&#39;)<br>    except ClientError as e:<br>        print(f&quot;Error ensuring log stream {log_stream_name}: {e}&quot;)<br>    return None<br><br>def create_batches(log_entries):<br>    &quot;&quot;&quot;Create batches of log events that adhere to CloudWatch limitations.&quot;&quot;&quot;<br>    batches = []<br>    batch = []<br>    batch_size = 0<br><br>    for log_entry in log_entries:<br>        try:<br>            match = ISO8601_TIMESTAMP_PATTERN.search(log_entry)<br>            if match:<br>                timestamp_str = match.group(0)<br>                log_time = datetime.strptime(timestamp_str, &quot;%Y-%m-%dT%H:%M:%S.%fZ&quot;)<br>            else:<br>                log_time = datetime.utcnow()<br><br>            event_time = int(log_time.timestamp() * 1000)<br>            log_event = {&#39;timestamp&#39;: event_time, &#39;message&#39;: log_entry}<br><br>            event_size = len(log_entry.encode(&#39;utf-8&#39;)) + LOG_EVENT_OVERHEAD<br>            if (batch_size + event_size) &gt; MAX_BATCH_SIZE or len(batch) &gt;= MAX_BATCH_COUNT:<br>                batches.append(batch)<br>                batch = []<br>                batch_size = 0<br><br>            batch.append(log_event)<br>            batch_size += event_size<br><br>        except Exception as e:<br>            print(f&quot;Failed to parse log entry &#39;{log_entry}&#39;: {e}&quot;)<br><br>    if batch:<br>        batches.append(batch)<br><br>    for batch in batches:<br>        batch.sort(key=lambda x: x[&#39;timestamp&#39;])<br><br>    return batches<br><br>def send_batches(log_group_name, log_stream_name, batches, sequence_token):<br>    &quot;&quot;&quot;Send log event batches to CloudWatch.&quot;&quot;&quot;<br>    for i, batch in enumerate(batches):<br>        sequence_token = put_events(log_group_name, log_stream_name, batch, sequence_token)<br>        if sequence_token is None:<br>            break<br><br>def put_events(log_group_name, log_stream_name, events, sequence_token):<br>    &quot;&quot;&quot;Put log events to a CloudWatch log stream.&quot;&quot;&quot;<br>    try:<br>        params = {<br>            &#39;logGroupName&#39;: log_group_name,<br>            &#39;logStreamName&#39;: log_stream_name,<br>            &#39;logEvents&#39;: events,<br>        }<br>        if sequence_token:<br>            params[&#39;sequenceToken&#39;] = sequence_token<br><br>        response = logs_client.put_log_events(**params)<br>        return response[&#39;nextSequenceToken&#39;]<br><br>    except logs_client.exceptions.InvalidSequenceTokenException as e:<br>        if &#39;sequenceToken&#39; in e.response[&#39;Error&#39;][&#39;Message&#39;]:<br>            new_token = e.response[&#39;Error&#39;][&#39;Message&#39;].split(&quot;The next expected sequenceToken is: &quot;)[1]<br>            return put_events(log_group_name, log_stream_name, events, new_token)<br>        else:<br>            return None<br>    except ClientError as e:<br>        print(f&quot;ClientError on put_log_events for {log_stream_name}: {e}&quot;)<br>        return None</pre><h3>Deploying the Solution</h3><p>To set up this solution, follow these steps:</p><ol><li><strong>Define the S3 bucket</strong> where logs will be stored.</li><li><strong>Write the Lambda function</strong> that processes S3 log events.</li><li><strong>Set up IAM permissions</strong> so Lambda can read from S3 and write to CloudWatch.</li><li><strong>Deploy the infrastructure using Terraform.</strong></li><li><strong>Monitor CloudWatch logs</strong> to ensure log ingestion is working correctly.</li></ol><h3>Conclusion</h3><p>If you enjoyed this article, don’t forget to come back for more.</p><p>By leveraging AWS Lambda to process S3 logs and publish them to CloudWatch, you create a more dynamic and responsive logging pipeline. This setup ensures that logs are available in real-time, enhancing observability, debugging, and alerting capabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2470d5b00c7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring AWS CloudWatch Log Activity with Terraform]]></title>
            <link>https://awstip.com/monitoring-aws-cloudwatch-log-activity-with-terraform-e08279138320?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/e08279138320</guid>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Mon, 10 Feb 2025 10:34:15 GMT</pubDate>
            <atom:updated>2025-02-10T18:02:36.608Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>When running applications in AWS, ensuring that logs are being written to CloudWatch is crucial for observability, debugging, and compliance. But what if your application stops logging? This could indicate an issue with the service, permissions, or even an application failure. To proactively detect such scenarios, we can set up an AWS CloudWatch alarm that notifies us when a log group is inactive.</p><p>In this guide, we’ll walk through setting up a CloudWatch alarm using Terraform to monitor whether logs are being written to a specific log group.</p><h3>Problem Statement</h3><p>You have a custom log group in AWS CloudWatch, and you want to monitor whether logs are continuously being written to it. If there are no incoming logs within a specific time frame, you want to be alerted.</p><h3>Solution Overview</h3><p>We will leverage AWS CloudWatch’s <strong>IncomingBytes</strong> metric, which tracks the volume of log data received by a log group. If this metric stays at zero for an extended period, we will trigger an alarm.</p><h3>Terraform Implementation</h3><p>Here’s a Terraform configuration to create a CloudWatch metric alarm that monitors incoming logs for a specific log group:</p><pre>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;log_inactivity_alarm&quot; {<br>  alarm_name          = &quot;AppLogInactivityAlarm&quot;<br>  comparison_operator = &quot;LessThanThreshold&quot;<br>  namespace           = &quot;AWS/Logs&quot;<br>  metric_name         = &quot;IncomingBytes&quot;<br>  dimensions = {<br>    LogGroupName = &quot;your-log-group-name&quot;<br>  }<br>  evaluation_periods  = 36  # 36 evaluation periods of 300 seconds each (3 hours total)<br>  period              = 300  # 5-minute periods<br>  statistic           = &quot;Sum&quot;<br>  threshold           = 1    # Alarm triggers if no logs are received in 3 hours<br>  alarm_description   = &quot;Triggers an alarm if logs are not received in the log group for over 3 hours.&quot;<br>  alarm_actions       = [aws_sns_topic.email_notifications.arn]<br>  treat_missing_data  = &quot;breaching&quot;<br>}</pre><h3>Breakdown of the Configuration</h3><ul><li><strong>alarm_name</strong> – Defines a human-readable name for the alarm.</li><li><strong>comparison_operator</strong> – Triggers the alarm when the metric falls below a defined threshold.</li><li><strong>namespace &amp; </strong><strong>metric_name</strong> – AWS CloudWatch Logs use the AWS/Logs namespace, and the metric of interest is IncomingBytes.</li><li><strong>dimensions</strong> – Specifies the log group to monitor.</li><li><strong>evaluation_periods &amp; </strong><strong>period</strong> – The alarm evaluates log activity in 5-minute windows and triggers if no activity is detected for 3 hours.</li><li><strong>threshold</strong> – The alarm is set to trigger if no bytes are logged.</li><li><strong>alarm_actions</strong> – Sends an alert via an SNS topic when the alarm is triggered.</li><li><strong>treat_missing_data = &quot;breaching&quot;</strong> – Ensures the alarm activates when no data is available.</li></ul><h3>Setting Up Alerts</h3><p>To receive notifications when the alarm is triggered, set up an AWS SNS (Simple Notification Service) topic:</p><pre>resource &quot;aws_sns_topic&quot; &quot;email_notifications&quot; {<br>  name = &quot;log-inactivity-alerts&quot;<br>}</pre><pre>resource &quot;aws_sns_topic_subscription&quot; &quot;email_sub&quot; {<br>  topic_arn = aws_sns_topic.email_notifications.arn<br>  protocol  = &quot;email&quot;<br>  endpoint  = &quot;your-email@example.com&quot;<br>}</pre><h3>Deploying the Configuration</h3><ol><li>Replace your-log-group-name with the actual log group you want to monitor.</li><li>Replace your-email@example.com with your actual email to receive notifications.</li><li>Run the following Terraform commands:</li></ol><pre>terraform init </pre><pre>terraform apply</pre><p>4. Confirm the SNS subscription by clicking the verification link sent to your email.</p><h3>Conclusion</h3><p>By implementing this Terraform-based solution, you can proactively monitor log activity in AWS CloudWatch and be alerted if logging stops unexpectedly. This helps maintain application observability, ensures compliance, and allows for quicker troubleshooting when issues arise.</p><p>Start monitoring your CloudWatch logs today and never miss critical application events again!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e08279138320" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/monitoring-aws-cloudwatch-log-activity-with-terraform-e08279138320">Monitoring AWS CloudWatch Log Activity with Terraform</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS CloudWatch Anomaly Detection via Terraform]]></title>
            <link>https://medium.com/@janngoncearuc/aws-cloudwatch-anomaly-detection-via-terraform-9bfd12f4b6d8?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bfd12f4b6d8</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 10:50:31 GMT</pubDate>
            <atom:updated>2025-01-31T11:11:28.959Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, a new resource has been introduced in the Terraform <strong>5.83.1</strong> release to enable anomaly detection on AWS <strong>CloudWatch</strong> log groups.</p><p>This feature allows you to identify unusual patterns in your logs, helping to monitor and debug your applications effectively.</p><p>To enable log anomaly detection on a log group in AWS, you need:</p><ol><li>The log group to already exist in AWS.</li><li>The corresponding Terraform configuration to manage it locally.</li></ol><p>Let’s assume we have an anomaly folder containing the following files:</p><ul><li>main.tf</li><li>locals.tf</li><li>variables.tf</li><li>alarms.tf</li></ul><h3>File Structure and Configuration</h3><h4>main.tf</h4><p>This file defines the core Terraform resources for enabling anomaly detection.</p><pre>resource &quot;aws_cloudwatch_log_anomaly_detector&quot; &quot;detector&quot; {<br>  for_each = {<br>    for lg in local.additional_log_groups :<br>    lg.name =&gt; {<br>      log_group         = data.aws_cloudwatch_log_group.log_groups[lg.name]<br>      evaluation_frequency   = lg.evaluation_frequency<br>      anomaly_visibility_time = lg.anomaly_visibility_time<br>      enabled                = lg.enabled<br>    }<br>  }<br><br>  detector_name           = each.value.log_group.name<br>  log_group_arn_list      = [each.value.log_group.arn]<br>  evaluation_frequency    = each.value.evaluation_frequency<br>  anomaly_visibility_time = each.value.anomaly_visibility_time<br>  enabled                 = tostring(each.value.enabled)<br>  tags                    = local.default_tags<br>  depends_on              = [aws_cloudwatch_log_group.alb_log_groups]<br>}<br><br>data &quot;aws_cloudwatch_log_group&quot; &quot;log_groups&quot; {<br>  for_each   = { for lg in local.additional_log_groups : lg.name =&gt; lg }<br>  name       = each.value.name<br>  depends_on = [aws_cloudwatch_log_group.alb_log_groups]<br>}</pre><h4>locals.tf</h4><p>This file contains the definitions for the log groups you want to activate anomaly detection on.</p><pre>locals {<br>  log_group_names = [for lg in local.additional_log_groups : lg.name]<br>  <br>  additional_log_groups = [<br>    {<br>      name                 = &quot;/test/log1&quot;<br>      evaluation_frequency = &quot;FIVE_MIN&quot;<br>      anomaly_visibility_time = 21<br>      enabled              = true<br>    },<br>    {<br>      name                 = &quot;/test/log2&quot;<br>      evaluation_frequency = &quot;FIVE_MIN&quot;<br>      anomaly_visibility_time = 12<br>      enabled              = true<br>    },<br>    {<br>      name                 = &quot;/test/log3&quot;<br>      evaluation_frequency = &quot;FIVE_MIN&quot;<br>      anomaly_visibility_time = 12<br>      enabled              = false<br>    }<br>  ]<br>}</pre><p><strong>alarms.tf</strong></p><p>This file contains the definitions for the <strong>HIGH</strong> alarms that you want to activate during the anomaly activation on log groups.</p><p>You can set as well alarm for <strong>MEDIUM</strong> anomalies by changing the dimension LogAnomalyPriority and of-course description.</p><pre>resource &quot;aws_sns_topic&quot; &quot;alerting&quot; {<br>  name = &quot;alerting-cahnnel&quot;<br>}<br><br><br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;high_anomaly_alarm&quot; {<br>  for_each = aws_cloudwatch_log_anomaly_detector.detector<br>  alarm_name          = &quot;HighAnomaly-${each.key}&quot;<br>  metric_name         = &quot;AnomalyCount&quot;<br>  namespace           = &quot;AWS/Logs&quot;<br>  statistic           = &quot;Sum&quot;<br>  period              = 300<br>  evaluation_periods  = 1<br>  datapoints_to_alarm = 1<br>  threshold           = 1<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  treat_missing_data  = &quot;missing&quot;<br><br>  alarm_description = &quot;Alarm definition for detecting anomalies in ${each.key}&quot;<br>  alarm_actions     = [aws_sns_topic.alerting.arn]<br><br>  dimensions = {<br>    LogAnomalyPriority  = &quot;HIGH&quot;<br>    LogAnomalyDetector = each.value.detector_name<br>  }<br>}</pre><h3>How to Apply</h3><p>After defining the configurations:</p><ol><li><strong>Initialize Terraform</strong>: Run terraform init to initialize your working directory.</li><li><strong>Apply the Configuration</strong>: Execute terraform apply to provision the resources.</li><li>At the end of the process, anomaly detection will be activated for the specified log groups.</li></ol><h3>Expected Outcome</h3><p>Once applied, you should see the anomaly detector activated and beginning to train on the specified log groups. This setup helps in monitoring application logs for irregular patterns and ensuring better visibility into the behavior of your system.</p><h3>Final Thoughts</h3><p>This approach showcases how Terraform can automate the setup of AWS CloudWatch anomaly detection, making it easier to maintain consistent configurations across environments.</p><p>Give it a try and let me know how it works for your use case!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bfd12f4b6d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hourly snapshots on AWS for Elastic Cloud via S]]></title>
            <link>https://medium.com/@janngoncearuc/hourly-snapshots-on-aws-for-elastic-cloud-via-s-b8d874a3dd55?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8d874a3dd55</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Wed, 22 Nov 2023 16:20:48 GMT</pubDate>
            <atom:updated>2023-11-22T16:21:37.471Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hourly snapshots on AWS for Elastic Cloud via S3 bucket</h3><p>In this article, I will teach you on how to enable hourly snapshots for Elastic Cloud cluster via AWS S3 bucket.</p><p>For this, we will need following resources to be in place:<br>- <strong>S3</strong> bucket: Storage that will keep track and dump of our Elastic SNAPSHOTS.<br>- <strong>IAM</strong> User: account that would have access only and only to this specific S3 bucket.<br>- <strong>Elastic</strong> Cluster: Where we will perform the policy itself. (deployment)<br>- <strong>KMS</strong> key: This key will be responsible of encrypting the data inside bucket itself; which means that you need to also harden the bucket ( make it private and so on).</p><p>IAM user should have the following policy attached to it:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Action&quot;: [<br>        &quot;*&quot;<br>      ],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:s3:::${S3_BUCKET}*&quot;,<br>        &quot;arn:aws:s3:::${S3_BUCKET}*/*&quot;<br>      ],<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Sid&quot;: &quot;FullAccess&quot;<br>    },<br>    {<br>      &quot;Sid&quot;: &quot;PartialAccess&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;kms:*&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;arn:aws:kms:${AWS_REGION}:${AWS_ACCOUNT_NUMBER}:key/*&quot;<br>    }<br>  ]<br>}</pre><p>Once the IAM user has been created, we need to access Elastic Cloud and navigate into our existing deployment and add security credentials that can be generated from IAM user created before that in previous step.</p><p>We took IAM security credentials for our user and we injected them into Elasticsearch keystore, which means that we have direct connection to AWS via this user and we have only and only access to the S3 bucket defined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b-ce2IggFdt3s3ut33ruiQ.png" /></figure><p>Now that user has been created, we need to create a bash script that would create policy for us inside Elastic Cluster:</p><pre>#!/bin/bash<br><br><br>log (){<br>  echo &quot;\n######## $1 ########\n&quot;<br>}<br><br>create_hourly_repository() {<br>     log &quot;Creating repository that would store the backup.&quot;<br>     curl -s --user ${elastic_user}:${elastic_password} -XPUT ${es_url}/_snapshot/${s3_bucket} -d\<br>     &#39;{<br>           &quot;type&quot;: &quot;s3&quot;,<br>           &quot;settings&quot;: {<br>             &quot;client&quot;: &quot;default&quot;,<br>             &quot;bucket&quot;: &quot;${s3_bucket}&quot;,<br>             &quot;region&quot;: &quot;${aws_region}&quot;<br>           }<br>         }&#39; \<br>     -H &#39;Accept: application/json&#39; -H &#39;Content-type: application/json&#39;<br>}<br><br><br>create_crontab_hourly_snapshot() {<br>   log &quot;Creating crontab with hourly snapshot dump into S3.&quot;<br>  curl -s --user ${elastic_user}:${elastic_password} -XPUT ${elastic_url}/_slm/policy/hourly-snapshots?pretty -H &#39;Content-Type: application/json&#39; -d&#39;<br>  {<br>    &quot;schedule&quot;: &quot;0 0 * * * ?&quot;,<br>    &quot;name&quot;: &quot;&lt;snapshot-{now/d{yyyy.MM.dd|+12:00}}&gt;&quot;,<br>    &quot;repository&quot;: &quot;${s3_bucket}&quot;,<br>    &quot;config&quot;: {<br>      &quot;indices&quot;: [&quot;*&quot;],<br>      &quot;ignore_unavailable&quot;: false,<br>      &quot;include_global_state&quot;: false<br>    },<br>    &quot;retention&quot;: {<br>      &quot;expire_after&quot;: &quot;30d&quot;,<br>      &quot;min_count&quot;: 20,<br>      &quot;max_count&quot;: 90<br>    }<br>  }&#39;<br>}<br><br><br>_main() {<br> create_hourly_repository<br> create_crontab_hourly_snapshot<br>}<br><br>_main &quot;$@&quot;<br><br></pre><p>Script legend:<br>1. <strong>elastic_user</strong>: Username used to login into elastic.<br>2. <strong>elastic_password</strong>: Password used to login to elastic.<br>3. <strong>elastic_url</strong>: Elastic URL.<br>4. <strong>aws_region</strong>: Region where the S3 bucket has been created.<br>5. <strong>s3_bucket</strong>: Bucket name that should be created as prerequisite.</p><p>Once script has been executed, we can navigate into Elastic Cloud under Snapshot area:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNaMfeFbgPsf_FxwgEGh5A.png" /><figcaption>Snapshot and Resotre section</figcaption></figure><p>If we will navigate into the Repositories section and we would verify repository, we will see that it’s connected to AWS and this allows us to perform hourly dump of our data as SNAPSHOST into S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gkg4YrUYtnAy9feqa4L2oA.png" /></figure><p>Hopefully you enjoyed this article! Stay tunned and follow me for future news.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8d874a3dd55" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elastic Cloud Cluster with Terraform via Private Link endpoint on AWS]]></title>
            <link>https://medium.com/@janngoncearuc/elastic-cloud-cluster-created-with-terraform-with-private-link-endpoint-between-55b802830df1?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b802830df1</guid>
            <category><![CDATA[elastic]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Thu, 12 Oct 2023 14:01:21 GMT</pubDate>
            <atom:updated>2023-10-14T19:19:31.711Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey there, lovely reader! Grab your coffee, tea, or energy drink (no judgment here), and plop down for a tale of epic scalability and easy-peasy deployment — starring Elastic Cloud and Terraform!</p><p>Think of Elastic Cloud as that friend we all have who’s super adaptable, always fitting in seamlessly, no matter the situation. It’s like that stretchy pair of pants in your wardrobe that accommodates your love for tacos without judgement. This cloud expands and contracts without a fuss, making sure your apps are running smooth and snappy, with room to breathe and grow.</p><p>Now, let’s bring Terraform into this tech party. Imagine Terraform as your savvy buddy who knows how to set up the entire home theater system with just a few flicks of the wrist (and without peeking at the manual). It’s the magical tool that lets you command the elastic powers of the Cloud with simple lines of code, turning what looks like a complex tech chore into a walk in the digital park.</p><p>In the world of cloud computing and data security, establishing a robust and secure network connection is paramount. That’s where Private Link comes into play.</p><p><strong>Private Link</strong> is a powerful feature that allows us to create a secure, private connection between our AWS infrastructure and Elastic Cloud. Unlike the public internet, Private Link ensures that your data stays within a controlled and isolated network environment, shielded from potential security risks and eavesdropping.</p><p>Key benefits of using Private Link in your Elasticsearch deployment include:</p><ol><li><strong>Enhanced Security</strong>: With Private Link, your Elasticsearch cluster is shielded from public access, reducing the surface area for potential threats. This added layer of security is crucial, especially when dealing with sensitive data.</li><li><strong>Improved Performance</strong>: Private Link provides low-latency, high-bandwidth connections between your AWS VPC (Virtual Private Cloud) and Elastic Cloud, ensuring that data can flow seamlessly and quickly.</li><li><strong>Isolation</strong>: Private Link isolates your Elasticsearch cluster from the public internet, making it an ideal choice for compliance-sensitive environments or applications that require strict access controls.</li><li><strong>Simplified Networking: </strong>Setting up Private Link simplifies the networking configuration, eliminating the need for complex VPNs or Direct Connect setups. It’s a straightforward way to establish a secure link between your AWS infrastructure and Elastic Cloud.</li></ol><p>In our step-by-step guide, we’ll walk you through the process of configuring Private Link to connect your AWS environment with Elastic Cloud, ensuring that your ElasticSearch data is not only powerful but also secure.</p><p>In order to start, we need to create following layout of files inside potential working directory.</p><p>Let’s assume that we are inside terraform directory and we want to create a deployment.<br>For this we will create first file named <strong>elastic.tf:</strong></p><pre>## We are creating Elastic cluster ##<br>resource &quot;ec_deployment&quot; &quot;deployment&quot; {<br>    name   = var.deployment_name<br>    region = var.aws_region<br><br>    version                = var.deployment_version<br>    deployment_template_id = var.deployment_template<br>    traffic_filter         = [ec_deployment_traffic_filter.deployment_filter.id]<br>    elasticsearch          = {<br>        hot = {<br>            zone_count  = var.elasticsearch_topology.zone_count<br>            size        = var.elasticsearch_topology.size<br>            autoscaling = {}<br>        }<br>    }<br><br><br>        kibana = {<br>            topology = {<br>                 size = var.kibana_topology.size<br>                }<br>            }<br>}<br><br>## Once cluster has been created, we are creating a traffic filter that will allow us to access it via private endpoints ##<br>## This will make cluster to deny all public requests ##<br><br>## Associating it with delpoyment ##<br>resource &quot;ec_deployment_traffic_filter_association&quot; &quot;deployment&quot; {<br>    traffic_filter_id = ec_deployment_traffic_filter.deployment_filter.id<br>    deployment_id     = ec_deployment.deployment.id<br>}<br><br>## We are defining the rule ##<br>resource &quot;ec_deployment_traffic_filter&quot; &quot;deployment_filter&quot; {<br>    name   = &quot;Allow traffic from AWS VPC for ${terraform.workspace}&quot;<br>    region = var.aws_region<br>    type   = &quot;vpce&quot;<br><br>    rule {<br>        source = aws_vpc_endpoint.private_link.id<br>    }<br></pre><p>Once we defined our resources for Elastic, we can create second file named <strong>network.tf</strong> that will have the following content:</p><pre><br>## We are creating Route53 Zone that will hold the entries or elastic itself ##<br>resource &quot;aws_route53_zone&quot; &quot;es_private_link&quot; {<br>  name          = lookup(var.elastic_zone_name, var.aws_region)<br>  tags          = var.tags<br>  vpc {<br>    vpc_id = var.vpc_id<br>  }<br>}<br><br>## We are adding record into our Route53 that will add mapping for our elastic deployment##<br><br>resource &quot;aws_route53_record&quot; &quot;private_link&quot; {<br>    zone_id = aws_route53_record.private_link.zone_id<br>    name    = &quot;test-elastic.es&quot;<br>    type    = &quot;CNAME&quot;<br>    records = [ lookup(aws_vpc_endpoint.private_link.dns_entry[ 0 ], &quot;dns_name&quot;) ]<br>    ttl = var.record_ttl<br>}<br><br><br>## Adding network that will allow our traffic ## <br>resource &quot;aws_security_group&quot; &quot;private_link&quot; {<br>    name   = var.aws_resource_name<br>    vpc_id = var.vpc_id<br><br>    ingress {<br>        description = &quot;Allow HTTPS to elastic cloud deployment&quot;<br>        from_port   = 443<br>        to_port     = 443<br>        protocol    = &quot;tcp&quot;<br>        cidr_blocks = var.ingress_cidr<br>    }<br><br>    tags = merge(<br>    {<br>        Name = &quot;${terraform.workspace}-${var.aws_resource_name}&quot;<br>    },<br>    var.tags,<br>    )<br>}<br><br>## Creating our Endpoint service that will connect AWS with Elastic ##<br>resource &quot;aws_vpc_endpoint&quot; &quot;private_link&quot; {<br>    service_name      = lookup(var.elastic_service_name, var.aws_region)<br>    vpc_id            = var.vpc_id<br>    vpc_endpoint_type = &quot;Interface&quot;<br>    security_group_ids = [ aws_security_group.private_link.id ]<br>    subnet_ids         = var.subnet_ids<br>    tags = merge(<br>    {<br>        Name = &quot;${terraform.workspace}-${var.aws_resource_name}&quot;<br>    },<br>    var.tags,<br>    )<br>    depends_on = [ aws_security_group.private_link ]<br><br>}</pre><p>Next file will be responsible for tiding up the providers (aws and elastic)</p><pre>terraform {<br>    required_version = &quot;&gt;= 1.0.0&quot;<br><br>    required_providers {<br>        ec = {<br>            source  = &quot;elastic/ec&quot;<br>            version = &quot;0.9.0&quot;<br>        }<br>    }<br><br>  required_providers {<br>      aws = {<br>        source  = &quot;hashicorp/aws&quot;<br>        version = &quot;5.20.0&quot;<br>      }<br>    }<br>}</pre><p>Next file will be <strong>vars.tf</strong> where we can have all our variables for provisioning of cluster:</p><pre>variable &quot;aws_resource_name&quot; {<br>    description = &quot;The name used for objects on AWS&quot;<br>    default     = &quot;elastic-cloud-private-link-access&quot;<br>}<br><br>variable &quot;vpc_id&quot; {<br>    description = &quot;VPC ID&quot;<br>}<br><br><br>/**<br>* Security Group<br>**/<br><br>variable &quot;ingress_cidr&quot; {<br>    description = &quot;CIDR Blocks that are allowed to connect to elastic cloud via the VPC&quot;<br>}<br><br>/**<br>* Private Link<br>**/<br><br><br>variable &quot;subnet_ids&quot; {<br>    description = &quot;ID&#39;s of the subnets to use for the VPCE. Be aware they MUST be in the supported AZ (https://www.elastic.co/guide/en/cloud/current/ec-traffic-filtering-vpc.html#ec-traffic-filtering-vpc)&quot;<br>}<br><br>/**<br>* Route53<br>**/<br><br>variable &quot;zone_name&quot; {<br>    description = &quot;Route53 zone name.&quot;<br>    default = {<br>        &quot;eu-west-1&quot;      = &quot;vpce.eu-west-1.aws.elastic-cloud.com&quot;<br>        &quot;us-east-1&quot;      = &quot;vpce.us-east-1.aws.elastic-cloud.com&quot;<br>    }<br>}<br><br>variable &quot;record_ttl&quot; {<br>    description = &quot;TTL for the Route53 Record. By default: 300&quot;<br>    default     = &quot;300&quot;<br>}<br><br>/**<br>* Elastic cloud deployment<br>**/<br><br>variable &quot;deployment_name&quot; {<br>    description = &quot;The name of the elastic cloud deployment&quot;<br>}<br><br>variable &quot;deployment_version&quot; {<br>    description = &quot;Version of the elastic deployment. (https://registry.terraform.io/providers/elastic/ec/latest/docs/data-sources/ec_stack)&quot;<br>}<br><br>variable &quot;deployment_template&quot; {<br>    description = &quot;Deployment template ID. (https://www.elastic.co/guide/en/cloud-enterprise/current/ece-deployment-templates.html)&quot;<br>    default     = &quot;aws-io-optimized-v2&quot;<br>}<br><br>variable &quot;elasticsearch_topology&quot; {<br>    description = &quot;Elasticsearch topology&quot;<br>    default     = {<br>        id         = &quot;hot_content&quot;<br>        zone_count = 2<br>        size       = &quot;4g&quot;<br>    }<br>}<br><br>variable &quot;kibana_topology&quot; {<br>    description = &quot;Kibana topology&quot;<br>    default     = {<br>        size = &quot;1g&quot;<br>    }<br>}<br>variable &quot;aws_region&quot; {<br>    description = &quot;AWS Region&quot;<br>}<br>variable &quot;tags&quot; {}<br><br><br>variable &quot;elastic_service_name&quot; {<br>    default = {<br>        &quot;eu-west-1&quot;      = &quot;com.amazonaws.vpce.eu-west-1.vpce-svc-01f2afe87944eb12b&quot;<br>        &quot;us-east-1&quot;      = &quot;com.amazonaws.vpce.us-east-1.vpce-svc-0e42e1e06ed010238&quot;<br>    }<br>}</pre><p>Now we can init terraform:</p><pre>terraform init </pre><p>Once all modules are initialised, we can apply Terraform.</p><p>IMPORTANT: In order to apply TF on elastic cloud, ensure that token has been created and has been exported as ENV variable in TF.<br>export EC_API_KEY=&quot;&lt;apikey value&gt;&quot;</p><pre>TF_WORKSPACE=testing terraform apply -var aws_region=us-east-1 -var aws_profile=poc-profile</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b802830df1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prowler on AWS via Terraform]]></title>
            <link>https://medium.com/@janngoncearuc/prowler-on-aws-via-terraform-b44fd3fddb9f?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/b44fd3fddb9f</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Fri, 07 Apr 2023 12:47:04 GMT</pubDate>
            <atom:updated>2023-04-07T12:47:54.154Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Ultimate Duo for Effective Cloud Security Management</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YA_Fr2E22E4lKXyizsgioA.jpeg" /></figure><p>Lately, I’ve been thinking about how to gather more information with less effort on incidents occurring in the AWS accounts that I work with on a daily basis. To be honest, verifying all the accounts (especially if you have multiple AWS accounts) for security incidents and implementing fixes can be a daunting task.</p><p>As a result, I started exploring different tools that could help me with this, and I became really excited about Prowler. Although it seemed challenging at first, as I added each piece to the bigger puzzle, it all started to make sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s9akrK2Ab-3vk4yb-IWmvQ.jpeg" /></figure><p>My first step was to create a Terraform layout with the following files</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*lQlGqBWf3A7be1F6Kbv-LQ.png" /></figure><p>1.<strong> </strong>The providers.tf file contains configuration details related to the provider (AWS), the region in which Terraform is being run, and the profile that should be used for running Terraform.</p><pre>provider &quot;aws&quot; {<br>  profile                  = var.aws_profile<br>  region                  = var.aws_region<br>}</pre><p>2.<strong> </strong>The data.tf file contains code that dynamically retrieves the current AWS region and the identity of the caller. This is achieved through the use of two data blocks: aws_region and aws_caller_identity.</p><pre>data &quot;aws_region&quot; &quot;current&quot; {}<br>data &quot;aws_caller_identity&quot; &quot;current&quot; {}</pre><p>3. The prowler.tf file contains code that defines all the resources related to Prowler, such as container properties. In this file, we can specify the format of the report that should be generated.</p><pre>data &quot;template_file&quot; &quot;prowler_container_properties&quot; {<br>  template = file(&quot;containers/prowler_container_properties.json&quot;)<br>  vars = {<br>    env_id                    = terraform.workspace<br>    region                    = var.aws_region<br>    registry_uri              = var.private_registry_url<br>    account                   = upper(terraform.workspace)<br>    stream_prefix             = &quot;${terraform.workspace}-security-scan-prowler&quot;<br>    report_format             = &quot;CSV&quot;<br>    execution_role_arn        = aws_iam_role.batch_execution_role.arn<br>    secret_name               = aws_secretsmanager_secret.docker_creds_secret.name<br>    secret_arn                = aws_secretsmanager_secret.docker_creds_secret.arn<br>  }<br>}<br><br>resource &quot;aws_batch_job_definition&quot; &quot;prowler_scan&quot; {<br>  name               = &quot;${terraform.workspace}-security-scan-prowler&quot;<br>  type               = &quot;container&quot;<br>  container_properties = data.template_file.prowler_container_properties.rendered<br>}<br><br># Cloudwatch rules and targets<br><br>resource &quot;aws_cloudwatch_event_rule&quot; &quot;prowler&quot; {<br>  name                = &quot;${terraform.workspace}-prowler-security-scan&quot;<br>  description         = &quot;Run prowler every day (23:15)&quot;<br>  schedule_expression = &quot;cron(15 23 * * ? *)&quot;<br>}<br><br>resource &quot;aws_cloudwatch_event_target&quot; &quot;prowler&quot; {<br>  rule     = aws_cloudwatch_event_rule.prowler.name<br>  arn      = aws_batch_job_queue.scan_queue.arn<br>  role_arn = aws_iam_role.cw_launch_security_scan.arn<br><br>  batch_target {<br>    job_name       = aws_batch_job_definition.prowler_scan.name<br>    job_definition = aws_batch_job_definition.prowler_scan.id<br>  }<br>}</pre><p>4. The variables.tf file contains the definition of all the variables related to the resources defined in the previous files.</p><pre>variable &quot;vpc_subnet_ids&quot;       {default = null}<br>variable &quot;vpc_id&quot;               {default = null}<br>variable &quot;aws_profile&quot;          {default = &quot;prowler-demo&quot;}<br>variable &quot;private_registry_url&quot; {default = &quot;docker.hub.demo&quot;}<br>variable &quot;aws_region&quot;           {default = &quot;eu-west-1&quot;}<br>variable &quot;tags&quot; {<br>  default = {<br>    &quot;created&quot;     = &quot;Automatically&quot;<br>    &quot;Iac Tool&quot;    = &quot;Terraform&quot;<br>  }<br>  description = &quot;Map of tags to append to resources&quot;<br>}</pre><p>5. The batch.tf file contains the code for running Prowler using the AWS Batch service, which runs serverless on ECS each time a CloudWatch event triggers a security scanning operation in the AWS account.</p><pre><br>data &quot;template_file&quot; &quot;scan_instance_policy_audit&quot; {<br>  template = file(<br>    &quot;iam_policies/security_scan_instance_policy_audit.json.tpl&quot;,<br>  )<br>  vars = {<br>    env_id     = terraform.workspace<br>    region     = var.aws_region<br>    account_id = data.aws_caller_identity.current.account_id<br>  }<br>}<br><br># IAM Instance role and Service Role<br>data &quot;template_file&quot; &quot;scan_instance_policy_audit_extra&quot; {<br>  template = file(<br>    &quot;iam_policies/security_scan_instance_policy_audit_extra.json.tpl&quot;,<br>  )<br>  vars = {<br>    env_id     = terraform.workspace<br>    region     = var.aws_region<br>    account_id = data.aws_caller_identity.current.account_id<br>  }<br>}<br><br>data &quot;template_file&quot; &quot;scan_instance_policy_ecs&quot; {<br>  template = file(&quot;iam_policies/security_scan_instance_policy_ecs.json.tpl&quot;)<br>  vars = {<br>    env_id     = terraform.workspace<br>    region     = var.aws_region<br>    account_id = data.aws_caller_identity.current.account_id<br>  }<br>}<br><br>data &quot;template_file&quot; &quot;scan_service_policy&quot; {<br>  template = file(&quot;iam_policies/security_scan_service_policy.json.tpl&quot;)<br>  vars = {<br>    env_id     = terraform.workspace<br>    region     = var.aws_region<br>    account_id = data.aws_caller_identity.current.account_id<br>  }<br>}<br><br>resource &quot;aws_iam_role&quot; &quot;batch_execution_role&quot; {<br>  name = &quot;batch_execution_role&quot;<br><br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Action = &quot;sts:AssumeRole&quot;<br>        Effect = &quot;Allow&quot;<br>        Principal = {<br>          Service = &quot;ecs-tasks.amazonaws.com&quot;<br>        }<br>      }<br>    ]<br>  })<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;batch_execution_role_policy&quot; {<br>  policy_arn = &quot;arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole&quot;<br>  role       = aws_iam_role.batch_execution_role.name<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;secrets_manager_access_policy&quot; {<br>  name = &quot;secrets_manager_access_policy&quot;<br><br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Action = [<br>          &quot;ecr:*&quot;<br>        ]<br>        Effect = &quot;Allow&quot;<br>        Resource = &quot;*&quot;<br>      }<br>    ]<br>  })<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;batch_policy&quot; {<br>  policy_arn = &quot;arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole&quot;<br>  role       = aws_iam_role.spot_service.name<br>}<br><br><br>resource &quot;aws_iam_role&quot; &quot;scan_instance&quot; {<br>  name               = &quot;${terraform.workspace}-security-scan-instance&quot;<br>  assume_role_policy = file(&quot;iam_policies/security_scan_instance_assume_role_policy.json&quot;)<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;scan_instance_audit&quot; {<br>  name   = &quot;${terraform.workspace}-security-scan-instance-audit&quot;<br>  policy = data.template_file.scan_instance_policy_audit.rendered<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;scan_instance_audit_extra&quot; {<br>  name   = &quot;${terraform.workspace}-security-scan-instance-audit-extra&quot;<br>  policy = data.template_file.scan_instance_policy_audit_extra.rendered<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;scan_instance_ecs&quot; {<br>  name   = &quot;${terraform.workspace}-security-scan-instance-ecs&quot;<br>  policy = data.template_file.scan_instance_policy_ecs.rendered<br>}<br><br>resource &quot;aws_iam_policy_attachment&quot; &quot;scan_instance_audit&quot; {<br>  name       = &quot;attachment-${terraform.workspace}-security-scan-instance-audit&quot;<br>  roles      = [aws_iam_role.scan_instance.name]<br>  policy_arn = aws_iam_policy.scan_instance_audit.arn<br>}<br><br>resource &quot;aws_iam_policy_attachment&quot; &quot;scan_instance_audit_extra&quot; {<br>  name       = &quot;attachment-${terraform.workspace}-security-scan-instance-audit-extra&quot;<br>  roles      = [aws_iam_role.scan_instance.name]<br>  policy_arn = aws_iam_policy.scan_instance_audit_extra.arn<br>}<br><br>resource &quot;aws_iam_policy_attachment&quot; &quot;scan_instance_ecs&quot; {<br>  name       = &quot;attachment-${terraform.workspace}-security-scan-instance-ecs&quot;<br>  roles      = [aws_iam_role.scan_instance.name]<br>  policy_arn = aws_iam_policy.scan_instance_ecs.arn<br>}<br><br>resource &quot;aws_iam_instance_profile&quot; &quot;scan_instance&quot; {<br>  name = &quot;${terraform.workspace}-security-scan&quot;<br>  role = aws_iam_role.scan_instance.name<br>}<br><br><br>resource &quot;aws_iam_role&quot; &quot;scan_service&quot; {<br>  name               = &quot;${terraform.workspace}-security-scan-service&quot;<br>  assume_role_policy = file(&quot;iam_policies/security_scan_service_assume_role_policy.json&quot;)<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;scan_service&quot; {<br>  name   = &quot;${terraform.workspace}-security-scan-service&quot;<br>  policy = data.template_file.scan_service_policy.rendered<br>}<br><br>resource &quot;aws_iam_policy_attachment&quot; &quot;scan_service&quot; {<br>  name       = &quot;attachment-${terraform.workspace}-security-scan-service&quot;<br>  roles      = [aws_iam_role.scan_service.name]<br>  policy_arn = aws_iam_policy.scan_service.arn<br>}<br><br>resource &quot;aws_iam_role&quot; &quot;spot_service&quot; {<br>  name               = &quot;${terraform.workspace}-security-scan-service-spotfleet&quot;<br>  assume_role_policy = file(&quot;iam_policies/security_scan_spot_fleet_assume_role_policy.json&quot;)<br>}<br><br># TODO: Create our own role<br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;spot_service_policy_attachment&quot; {<br>  role       = aws_iam_role.spot_service.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole&quot;<br>}<br><br>resource &quot;aws_security_group&quot; &quot;scan&quot; {<br>  name   = &quot;${terraform.workspace}-batch-scan&quot;<br>  vpc_id = var.vpc_id<br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>}<br>data &quot;template_file&quot; &quot;userdata&quot; {<br>  template = file(&quot;${path.cwd}/files/docker-login.sh&quot;)<br>  vars = {<br>    secret = aws_secretsmanager_secret.docker_creds_secret.name<br>  }<br>}<br><br>resource &quot;aws_launch_template&quot; &quot;scan_batch_template&quot; {<br>  name = &quot;${terraform.workspace}-scan-batch-template&quot;<br>  user_data = base64encode(data.template_file.userdata.rendered)<br>}<br><br><br># AWS Batch - Compute environment, queue and job<br>resource &quot;aws_batch_compute_environment&quot; &quot;scan&quot; {<br>  compute_environment_name = &quot;${terraform.workspace}-security-scan-compute&quot;<br><br>  compute_resources {<br>    instance_role = aws_iam_instance_profile.scan_instance.arn<br>    max_vcpus     = 8<br>    min_vcpus     = 0<br>    instance_type = [<br>      &quot;m5.large&quot;,<br>    ]<br>    security_group_ids = [<br>      aws_security_group.scan.id<br>    ]<br>    subnets             = var.vpc_subnet_ids<br>    type                = &quot;SPOT&quot;<br>    spot_iam_fleet_role = aws_iam_role.spot_service.arn<br>    bid_percentage      = &quot;80&quot;<br>    launch_template {<br>      launch_template_id = aws_launch_template.scan_batch_template.id<br>    }<br>  }<br><br>  service_role = aws_iam_role.scan_service.arn<br>  type         = &quot;MANAGED&quot;<br>  depends_on   = [aws_iam_policy_attachment.scan_service]<br>}<br><br>resource &quot;aws_batch_job_queue&quot; &quot;scan_queue&quot; {<br>  name                 = &quot;${terraform.workspace}-security-scan-queue&quot;<br>  state                = &quot;ENABLED&quot;<br>  priority             = 1<br>  compute_environments = [aws_batch_compute_environment.scan.arn]<br>}<br><br>## Secrets ##<br>resource &quot;aws_secretsmanager_secret&quot; &quot;docker_creds_secret&quot; {<br>  name = &quot;registry_creds_secret&quot;<br>}<br><br>resource &quot;aws_secretsmanager_secret_version&quot; &quot;docker_creds_secret_version&quot; {<br>  secret_id     = aws_secretsmanager_secretdocker_creds_secret.id<br>  secret_string = jsonencode({<br>    &quot;username&quot;      = &quot;USERNAME&quot;,<br>    &quot;password&quot;      = &quot;PASSWORD&quot;,<br>    &quot;registry_url&quot;  = var.private_registry_url<br>  })<br>}<br><br></pre><p>6. The cloudwatch.tf file contains code that defines the CloudWatch role along with its policy.</p><pre># IAM Role for Cloud Watch to Launch Security Scans<br>data &quot;template_file&quot; &quot;cw_launch_security_scan_policy&quot; {<br>  template = file(&quot;iam_policies/cloudwatch_launch_security_scan_policy.json.tpl&quot;)<br>  vars = {<br>    env_id     = terraform.workspace<br>    region     = var.aws_region<br>    account_id = data.aws_caller_identity.current.account_id<br>    queue      = aws_batch_job_queue.scan_queue.name<br>  }<br>}<br><br>resource &quot;aws_iam_role&quot; &quot;cw_launch_security_scan&quot; {<br>  name               = &quot;${terraform.workspace}-cw-launch-security-scan&quot;<br>  assume_role_policy = file(&quot;iam_policies/cloudwatch_launch_security_scan_assume_role_policy.json&quot;)<br>}<br><br>resource &quot;aws_iam_policy&quot; &quot;cw_launch_security_scan&quot; {<br>  name   = &quot;${terraform.workspace}-cw-launch-scan-service&quot;<br>  policy = data.template_file.cw_launch_security_scan_policy.rendered<br>}<br><br>resource &quot;aws_iam_policy_attachment&quot; &quot;cw_launch_security_scan&quot; {<br>  name       = &quot;${terraform.workspace}-attachment-cw-launch-security-scan&quot;<br>  roles      = [aws_iam_role.cw_launch_security_scan.name]<br>  policy_arn = aws_iam_policy.cw_launch_security_scan.arn<br>}<br></pre><p>7. In this file, we define the Terraform backend that will store the state of our infrastructure. The backend block specifies that we will be using the S3 backend, with the bucket name, key, and region specified.</p><p>Note that the specific values used in this example will need to be modified to match your own S3 bucket, region, and other requirements.</p><pre><br>terraform {<br>  backend &quot;s3&quot; {<br>    bucket       = &quot;terraform-state&quot;<br>    key          = &quot;demo-prowler&quot;<br>    region       = &quot;eu-west-1&quot;<br>    session_name = &quot;terraform&quot;<br>    profile       = &quot;demo-prowler&quot;<br>  }<br>}</pre><p>In this file, we specify that we want to store our Terraform state remotely in a bucket named terraform-state, with the identifier demo-prowler used as the key.</p><p>Now it’s time to create container/prowler_container_properties.json</p><p>file that will contain the properties of our ECS fleet :</p><pre>{<br>    &quot;image&quot;: &quot;${image}&quot;,<br>    &quot;memory&quot;: 2048,<br>    &quot;vcpus&quot;: 2,<br>    &quot;executionRoleArn&quot; : &quot;${execution_role_arn}&quot;,<br>    &quot;secrets&quot;: [<br>        {<br>            &quot;name&quot;: &quot;${secret_name}&quot;,<br>            &quot;valueFrom&quot;: &quot;${secret_arn}&quot;<br>        }<br>    ],<br>    &quot;environment&quot;: [<br>        {<br>            &quot;name&quot;: &quot;ACCOUNT&quot;,<br>            &quot;value&quot;: &quot;${account}&quot;<br>        },<br>        {<br>            &quot;name&quot;: &quot;FUNCTION_REGION&quot;,<br>            &quot;value&quot;: &quot;${region}&quot;<br>        },<br>        {<br>            &quot;name&quot;: &quot;STREAM_PREFIX&quot;,<br>            &quot;value&quot;: &quot;${stream_prefix}&quot;<br>        },<br>        {<br>            &quot;name&quot;: &quot;REPORT_FORMAT&quot;,<br>            &quot;value&quot;: &quot;${report_format}&quot;<br>        },<br>        {<br>            &quot;name&quot;: &quot;REGISTRY_URL&quot;,<br>            &quot;value&quot;: &quot;${registry_uri}&quot;<br>        },<br>        {<br>            &quot;name&quot;: &quot;SECRET_NAME&quot;,<br>            &quot;value&quot;: &quot;${secret_name}&quot;<br>        }<br>    ]<br>}</pre><p>Once we have created all the .tf files, we need to ensure that we have the necessary policies that can be read and fetched by Terraform&#39;s data resources. To do this, create a folder named iam_policies in the current Terraform directory, and ensure that it contains the following files:</p><p>A. security_scan_instance_policy_audit.json.tpl: This file grants permissions to the Security Scan instance to check all audit events stored in services such as CloudTrail, Config, Analyser, and so on.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Action&quot;: [<br>                &quot;access-analyzer:Get*&quot;,<br>                &quot;access-analyzer:List*&quot;,<br>                &quot;acm:Describe*&quot;,<br>                &quot;acm:List*&quot;,<br>                &quot;application-autoscaling:Describe*&quot;,<br>                &quot;ecr:GetAuthorizationToken&quot;,<br>                &quot;autoscaling:Describe*&quot;,<br>                &quot;batch:DescribeComputeEnvironments&quot;,<br>                &quot;batch:DescribeJobDefinitions&quot;,<br>                &quot;chime:List*&quot;,<br>                &quot;cloudformation:List*&quot;,<br>                &quot;cloudfront:Get*&quot;,<br>                &quot;cloudfront:List*&quot;,<br>                &quot;cloudtrail:DescribeTrails&quot;,<br>                &quot;cloudtrail:GetEventSelectors&quot;,<br>                &quot;cloudtrail:GetTrailStatus&quot;,<br>                &quot;cloudtrail:ListTags&quot;,<br>                &quot;cloudtrail:LookupEvents&quot;,<br>                &quot;cloudwatch:Describe*&quot;,<br>                &quot;cognito-identity:ListIdentityPools&quot;,<br>                &quot;cognito-idp:ListUserPools&quot;,<br>                &quot;cognito-sync:Describe*&quot;,<br>                &quot;cognito-sync:List*&quot;,<br>                &quot;config:BatchGetAggregateResourceConfig&quot;,<br>                &quot;config:BatchGetResourceConfig&quot;,<br>                &quot;config:Deliver*&quot;,<br>                &quot;config:Describe*&quot;,<br>                &quot;config:Get*&quot;,<br>                &quot;config:List*&quot;,<br>                &quot;datapipeline:ListPipelines&quot;,<br>                &quot;datasync:Describe*&quot;,<br>                &quot;datasync:List*&quot;,<br>                &quot;directconnect:Describe*&quot;,<br>                &quot;dms:Describe*&quot;,<br>                &quot;dms:ListTagsForResource&quot;,<br>                &quot;ds:DescribeDirectories&quot;,<br>                &quot;dynamodb:Describe*&quot;,<br>                &quot;dynamodb:List*&quot;,<br>                &quot;ec2:Describe*&quot;,<br>                &quot;ecr:DescribeRepositories&quot;,<br>                &quot;ecr:GetRepositoryPolicy&quot;,<br>                &quot;ecs:Describe*&quot;,<br>                &quot;ecs:List*&quot;,<br>                &quot;eks:DescribeCluster&quot;,<br>                &quot;eks:ListClusters&quot;,<br>                &quot;elasticache:Describe*&quot;,<br>                &quot;elasticbeanstalk:Describe*&quot;,<br>                &quot;elasticfilesystem:Describe*&quot;,<br>                &quot;elasticloadbalancing:Describe*&quot;,<br>                &quot;es:Describe*&quot;,<br>                &quot;es:ListDomainNames&quot;,<br>                &quot;events:Describe*&quot;,<br>                &quot;events:List*&quot;,<br>                &quot;firehose:Describe*&quot;,<br>                &quot;firehose:List*&quot;,<br>                &quot;fms:ListComplianceStatus&quot;,<br>                &quot;fms:ListPolicies&quot;,<br>                &quot;fsx:Describe*&quot;,<br>                &quot;fsx:List*&quot;,<br>                &quot;glacier:DescribeVault&quot;,<br>                &quot;glacier:GetVaultAccessPolicy&quot;,<br>                &quot;glacier:ListVaults&quot;,<br>                &quot;guardduty:Get*&quot;,<br>                &quot;guardduty:List*&quot;,<br>                &quot;iam:GenerateCredentialReport&quot;,<br>                &quot;iam:GenerateServiceLastAccessedDetails&quot;,<br>                &quot;iam:Get*&quot;,<br>                &quot;iam:List*&quot;,<br>                &quot;iam:SimulateCustomPolicy&quot;,<br>                &quot;iam:SimulatePrincipalPolicy&quot;,<br>                &quot;inspector:Describe*&quot;,<br>                &quot;inspector:Get*&quot;,<br>                &quot;inspector:List*&quot;,<br>                &quot;inspector:Preview*&quot;,<br>                &quot;iot:Describe*&quot;,<br>                &quot;iot:Get*&quot;,<br>                &quot;iot:List*&quot;,<br>                &quot;kinesis:DescribeStream&quot;,<br>                &quot;kinesis:ListStreams&quot;,<br>                &quot;kinesis:ListTagsForStream&quot;,<br>                &quot;kinesisanalytics:ListApplications&quot;,<br>                &quot;kms:Describe*&quot;,<br>                &quot;kms:Get*&quot;,<br>                &quot;kms:List*&quot;,<br>                &quot;lambda:Get*&quot;,<br>                &quot;lambda:List*&quot;,<br>                &quot;license-manager:List*&quot;,<br>                &quot;logs:Describe*&quot;,<br>                &quot;logs:ListTagsLogGroup&quot;,<br>                &quot;ram:List*&quot;,<br>                &quot;rds:Describe*&quot;,<br>                &quot;rds:DownloadDBLogFilePortion&quot;,<br>                &quot;rds:ListTagsForResource&quot;,<br>                &quot;redshift:Describe*&quot;,<br>                &quot;route53:Get*&quot;,<br>                &quot;route53:List*&quot;,<br>                &quot;route53domains:GetDomainDetail&quot;,<br>                &quot;route53domains:GetOperationDetail&quot;,<br>                &quot;route53domains:ListDomains&quot;,<br>                &quot;route53domains:ListOperations&quot;,<br>                &quot;route53domains:ListTagsForDomain&quot;,<br>                &quot;route53resolver:List*&quot;,<br>                &quot;route53resolver:Get*&quot;,<br>                &quot;s3:Get*&quot;,<br>                &quot;s3:ListAccessPoints&quot;,<br>                &quot;s3:ListAllMyBuckets&quot;,<br>                &quot;sdb:DomainMetadata&quot;,<br>                &quot;sdb:ListDomains&quot;,<br>                &quot;secretsmanager:GetResourcePolicy&quot;,<br>                &quot;secretsmanager:ListSecrets&quot;,<br>                &quot;secretsmanager:ListSecretVersionIds&quot;,<br>                &quot;securityhub:Describe*&quot;,<br>                &quot;securityhub:Get*&quot;,<br>                &quot;securityhub:List*&quot;,<br>                &quot;serverlessrepo:GetApplicationPolicy&quot;,<br>                &quot;serverlessrepo:List*&quot;,<br>                &quot;ses:GetIdentityDkimAttributes&quot;,<br>                &quot;ses:GetIdentityPolicies&quot;,<br>                &quot;ses:GetIdentityVerificationAttributes&quot;,<br>                &quot;ses:List*&quot;,<br>                &quot;shield:Describe*&quot;,<br>                &quot;shield:List*&quot;,<br>                &quot;snowball:ListClusters&quot;,<br>                &quot;snowball:ListJobs&quot;,<br>                &quot;sns:GetTopicAttributes&quot;,<br>                &quot;sns:List*&quot;,<br>                &quot;sqs:GetQueueAttributes&quot;,<br>                &quot;sqs:List*&quot;,<br>                &quot;sso:DescribePermissionsPolicies&quot;,<br>                &quot;sso:List*&quot;,<br>                &quot;states:ListStateMachines&quot;,<br>                &quot;storagegateway:Describe*&quot;,<br>                &quot;storagegateway:List*&quot;,<br>                &quot;tag:GetResources&quot;,<br>                &quot;tag:GetTagKeys&quot;,<br>                &quot;transfer:Describe*&quot;,<br>                &quot;transfer:List*&quot;,<br>                &quot;trustedadvisor:Describe*&quot;,<br>                &quot;waf:ListWebACLs&quot;,<br>                &quot;waf-regional:ListWebACLs&quot;,<br>                &quot;workspaces:Describe*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;apigateway:GET&quot;<br>            ],<br>     &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><p>B. security_scan_instance_policy_audit_extra.json.tpl grants additional permissions to Prowler for running in the AWS account.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br> {<br>          &quot;Action&quot;: [<br>              &quot;access-analyzer:List*&quot;,<br>              &quot;apigateway:get*&quot;,<br>              &quot;apigatewayv2:get*&quot;,<br>              &quot;aws-marketplace:viewsubscriptions&quot;,<br>              &quot;batch:listjobs&quot;,<br>              &quot;clouddirectory:listappliedschemaarns&quot;,<br>              &quot;clouddirectory:listdevelopmentschemaarns&quot;,<br>              &quot;clouddirectory:listpublishedschemaarns&quot;,<br>              &quot;cloudformation:list*&quot;,<br>              &quot;cloudhsm:listavailablezones&quot;,<br>              &quot;cloudsearch:list*&quot;,<br>              &quot;cloudwatch:get*&quot;,<br>              &quot;cloudwatch:list*&quot;,<br>              &quot;codebuild:listbuilds*&quot;,<br>              &quot;codestar:verify*&quot;,<br>              &quot;cognito-identity:listidentities&quot;,<br>              &quot;cognito-idp:list*&quot;,<br>              &quot;cognito-sync:listdatasets&quot;,<br>              &quot;connect:list*&quot;,<br>              &quot;datapipeline:getaccountlimits&quot;,<br>              &quot;dax:describeclusters&quot;,<br>              &quot;dax:describedefaultparameters&quot;,<br>              &quot;dax:describeevents&quot;,<br>              &quot;dax:describeparametergroups&quot;,<br>              &quot;dax:describeparameters&quot;,<br>              &quot;dax:describesubnetgroups&quot;,<br>              &quot;dax:describetable&quot;,<br>              &quot;dax:listtables&quot;, <br>              &quot;devicefarm:list*&quot;,<br>              &quot;discovery:list*&quot;,<br>              &quot;dms:list*&quot;,<br>              &quot;ds:ListAuthorizedApplications&quot;,<br>              &quot;ds:DescribeRoles&quot;,<br>              &quot;dynamodb:describebackup&quot;,<br>              &quot;dynamodb:describeglobaltablesettings&quot;,<br>              &quot;dynamodb:describelimits&quot;,<br>              &quot;dynamodb:describereservedcapacity&quot;,<br>              &quot;dynamodb:describereservedcapacityofferings&quot;,<br>              &quot;dynamodb:describestream&quot;,<br>              &quot;dynamodb:listtagsofresource&quot;,<br>              &quot;dynamodb:scan&quot;,<br>              &quot;dynamodb:query&quot;,<br>              &quot;ec2:get*&quot;,<br>              &quot;ecr:describe*&quot;,<br>              &quot;ecr:listimages&quot;,<br>              &quot;elasticbeanstalk:listavailablesolutionstacks&quot;,<br>              &quot;elasticmapreduce:list*&quot;,<br>              &quot;elastictranscoder:list*&quot;,<br>              &quot;gamelift:list*&quot;,<br>              &quot;glacier:list*&quot;,<br>              &quot;importexport:listjobs&quot;,<br>              &quot;lambda:GetAccountSettings&quot;,<br>              &quot;lambda:GetFunctionConfiguration&quot;,<br>              &quot;lambda:GetLayerVersionPolicy&quot;,<br>              &quot;lambda:GetPolicy&quot;,<br>              &quot;lambda:List*&quot;,<br>              &quot;lex:getbotaliases&quot;,<br>              &quot;lex:getbotchannelassociations&quot;,<br>              &quot;lex:getbots&quot;,<br>              &quot;lex:getbotversions&quot;,<br>              &quot;lex:getintents&quot;,<br>              &quot;lex:getintentversions&quot;,<br>              &quot;lex:getslottypes&quot;,<br>              &quot;lex:getslottypeversions&quot;,<br>              &quot;lex:getutterancesview&quot;,<br>              &quot;lightsail:getblueprints&quot;,<br>              &quot;lightsail:getbundles&quot;,<br>              &quot;lightsail:getinstancesnapshots&quot;,<br>              &quot;lightsail:getkeypair&quot;,<br>              &quot;lightsail:getregions&quot;,<br>              &quot;lightsail:getstaticips&quot;,<br>              &quot;lightsail:isvpcpeered&quot;,<br>              &quot;machinelearning:describe*&quot;,<br>              &quot;mobilehub:listavailablefeatures&quot;,<br>              &quot;mobilehub:listavailableregions&quot;,<br>              &quot;mobilehub:listprojects&quot;,<br>              &quot;mobiletargeting:getapplicationsettings&quot;,<br>              &quot;mobiletargeting:getcampaigns&quot;,<br>              &quot;mobiletargeting:getimportjobs&quot;,<br>              &quot;mobiletargeting:getsegments&quot;,<br>              &quot;opsworks-cm:describe*&quot;,<br>              &quot;opsworks:describe*&quot;,<br>              &quot;polly:describe*&quot;,<br>              &quot;polly:list*&quot;,<br>              &quot;redshift:viewqueriesinconsole&quot;,<br>              &quot;route53domains:list*&quot;,<br>              &quot;s3:listbucket&quot;,<br>              &quot;sdb:list*&quot;,<br>              &quot;secretsmanager:listsecretversionids&quot;,<br>              &quot;servicecatalog:list*&quot;,<br>              &quot;ses:list*&quot;,<br>              &quot;sns:list*&quot;,<br>              &quot;sqs:listqueuetags&quot;,<br>              &quot;ssm:listassociations&quot;,<br>              &quot;states:listactivities&quot;,<br>              &quot;support:describe*&quot;,<br>              &quot;swf:list*&quot;,<br>              &quot;tag:gettagkeys&quot;,<br>              &quot;trustedadvisor:describe*&quot;,<br>              &quot;waf-regional:list*&quot;,<br>              &quot;waf:list*&quot;,<br>              &quot;workdocs:describeavailabledirectories&quot;,<br>              &quot;workdocs:describeinstances&quot;,<br>              &quot;workmail:describe*&quot;,<br>              &quot;ecr:GetAuthorizationToken&quot;<br>          ],<br>          &quot;Effect&quot;: &quot;Allow&quot;,<br>          &quot;Resource&quot;: &quot;*&quot;<br>       }<br>    ]<br>}</pre><p>4. security_scan_instance_policy_ecs.json.tpl — Adding permissions for ECS to have the ability to run.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:DescribeTags&quot;,<br>                &quot;ecs:CreateCluster&quot;,<br>                &quot;ecs:DeregisterContainerInstance&quot;,<br>                &quot;ecs:DiscoverPollEndpoint&quot;,<br>                &quot;ecs:Poll&quot;,<br>                &quot;ecs:RegisterContainerInstance&quot;,<br>                &quot;ecs:StartTelemetrySession&quot;,<br>                &quot;ecs:UpdateContainerInstancesState&quot;,<br>                &quot;ecs:Submit*&quot;,<br>                &quot;ecr:GetAuthorizationToken&quot;,<br>                &quot;ecr:BatchCheckLayerAvailability&quot;,<br>                &quot;ecr:GetDownloadUrlForLayer&quot;,<br>                &quot;ecr:BatchGetImage&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><p>C. security_scan_instance_assume_role_policy.json defines the role that will be assumed by instances for security scanning.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>    {<br>        &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>        &quot;Effect&quot;: &quot;Allow&quot;,<br>        &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;ec2.amazonaws.com&quot;<br>        }<br>    }<br>    ]<br>}</pre><p>C. security_scan_service_policy.json.tpl defines the service policy for security scanning.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:DescribeAccountAttributes&quot;,<br>                &quot;ec2:DescribeInstances&quot;,<br>                &quot;ec2:DescribeInstanceAttribute&quot;,<br>                &quot;ec2:DescribeSubnets&quot;,<br>                &quot;ec2:DescribeSecurityGroups&quot;,<br>                &quot;ec2:DescribeKeyPairs&quot;,<br>                &quot;ec2:DescribeImages&quot;,<br>                &quot;ec2:DescribeImageAttribute&quot;,<br>                &quot;ec2:DescribeSpotInstanceRequests&quot;,<br>                &quot;ec2:DescribeSpotFleetInstances&quot;,<br>                &quot;ec2:DescribeSpotFleetRequests&quot;,<br>                &quot;ec2:DescribeSpotPriceHistory&quot;,<br>                &quot;ec2:DescribeVpcClassicLink&quot;,<br>                &quot;ec2:DescribeLaunchTemplateVersions&quot;,<br>                &quot;ec2:CreateLaunchTemplate&quot;,<br>                &quot;ec2:DeleteLaunchTemplate&quot;,<br>                &quot;ec2:RequestSpotFleet&quot;,<br>                &quot;ec2:CancelSpotFleetRequests&quot;,<br>                &quot;ec2:ModifySpotFleetRequest&quot;,<br>                &quot;ec2:TerminateInstances&quot;,<br>                &quot;ec2:RunInstances&quot;,<br>                &quot;autoscaling:DescribeAccountLimits&quot;,<br>                &quot;autoscaling:DescribeAutoScalingGroups&quot;,<br>                &quot;autoscaling:DescribeLaunchConfigurations&quot;,<br>                &quot;autoscaling:DescribeAutoScalingInstances&quot;,<br>                &quot;autoscaling:CreateLaunchConfiguration&quot;,<br>                &quot;autoscaling:CreateAutoScalingGroup&quot;,<br>                &quot;autoscaling:UpdateAutoScalingGroup&quot;,<br>                &quot;autoscaling:SetDesiredCapacity&quot;,<br>                &quot;autoscaling:DeleteLaunchConfiguration&quot;,<br>                &quot;autoscaling:DeleteAutoScalingGroup&quot;,<br>                &quot;autoscaling:CreateOrUpdateTags&quot;,<br>                &quot;autoscaling:SuspendProcesses&quot;,<br>                &quot;autoscaling:PutNotificationConfiguration&quot;,<br>                &quot;autoscaling:TerminateInstanceInAutoScalingGroup&quot;,<br>                &quot;ecs:DescribeClusters&quot;,<br>                &quot;ecs:DescribeContainerInstances&quot;,<br>                &quot;ecs:DescribeTaskDefinition&quot;,<br>                &quot;ecs:DescribeTasks&quot;,<br>                &quot;ecs:ListClusters&quot;,<br>                &quot;ecs:ListContainerInstances&quot;,<br>                &quot;ecs:ListTaskDefinitionFamilies&quot;,<br>                &quot;ecs:ListTaskDefinitions&quot;,<br>                &quot;ecs:ListTasks&quot;,<br>                &quot;ecs:CreateCluster&quot;,<br>                &quot;ecs:DeleteCluster&quot;,<br>                &quot;ecs:RegisterTaskDefinition&quot;,<br>                &quot;ecs:DeregisterTaskDefinition&quot;,<br>                &quot;ecs:RunTask&quot;,<br>                &quot;ecs:StartTask&quot;,<br>                &quot;ecs:StopTask&quot;,<br>                &quot;ecs:UpdateContainerAgent&quot;,<br>                &quot;ecs:DeregisterContainerInstance&quot;,<br>                &quot;ecr:GetAuthorizationToken&quot;,<br>                &quot;logs:CreateLogGroup&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;,<br>                &quot;logs:DescribeLogGroups&quot;,<br>                &quot;iam:GetInstanceProfile&quot;,<br>                &quot;iam:GetRole&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;iam:PassRole&quot;,<br>            &quot;Resource&quot;: [<br>                &quot;*&quot;<br>            ],<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;iam:PassedToService&quot;: [<br>                        &quot;ec2.amazonaws.com&quot;,<br>                        &quot;ec2.amazonaws.com.cn&quot;,<br>                        &quot;ecs-tasks.amazonaws.com&quot;<br>                    ]<br>                }<br>            }<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;iam:CreateServiceLinkedRole&quot;,<br>            &quot;Resource&quot;: &quot;*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;iam:AWSServiceName&quot;: [<br>                        &quot;spot.amazonaws.com&quot;,<br>                        &quot;spotfleet.amazonaws.com&quot;,<br>                        &quot;autoscaling.amazonaws.com&quot;,<br>                        &quot;ecs.amazonaws.com&quot;<br>                    ]<br>                }<br>            }<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:CreateTags&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;*&quot;<br>            ],<br>            &quot;Condition&quot;: {<br>                &quot;StringEquals&quot;: {<br>                    &quot;ec2:CreateAction&quot;: &quot;RunInstances&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><p>With our Prowler configuration almost complete, we need to add the final elements to Terraform. In the main layout, make sure you have also created a folder named files that contains a script parsed to ECS for authenticating into a registry, in case you use a private Docker registry.</p><p>Add the following script under <strong>files</strong>/docker-login.sh:</p><pre>MIME-Version: 1.0<br>Content-Type: multipart/mixed; boundary=&quot;==MYBOUNDARY==&quot;<br><br>--==MYBOUNDARY==<br>Content-Type: text/cloud-config; charset=&quot;us-ascii&quot;<br><br>packages:<br><br>- jq<br>- aws-cli<br>runcmd:<br><br>/usr/bin/aws configure set region $(curl http://169.254.169.254/latest/meta-data/placement/region)<br>export SECRET_STRING=$(/usr/bin/aws secretsmanager get-secret-value --secret-id ${secret} | jq -r &#39;.SecretString&#39;)<br>export DOCKER_USERNAME=$(echo $SECRET_STRING | jq -r &#39;.docker_username&#39;)<br>export DOCKER_PASSWORD=$(echo $SECRET_STRING | jq -r &#39;.docker_password&#39;)<br>docker login --username $DOCKER_USERNAME --password $DOCKER_PASSWORD<br>export AUTH=$(cat ~/.docker/config.json | jq -c .auths)<br>echo &quot;ECS_ENGINE_AUTH_TYPE=dockercfg&quot; &gt;&gt; /etc/ecs/ecs.config<br>echo &quot;ECS_ENGINE_AUTH_DATA=$AUTH&quot; &gt;&gt; /etc/ecs/ecs.config<br>--==MYBOUNDARY==--</pre><p>In conclusion, if you’re looking for a reliable and efficient way to manage the security of your AWS resources, Terraform combined with Prowler on AWS is an excellent solution. In this discussion, we covered the essential files needed to set up a Prowler security scanning environment on AWS using Terraform.</p><p>By leveraging the power of Terraform to define infrastructure as code, we can easily manage and automate the provisioning of cloud resources while ensuring security and compliance with industry standards.</p><p>With the use of Prowler, we can perform automated security scans and receive alerts in real-time to help us stay ahead of potential security issues.</p><p>If you’re new to Terraform and Prowler, getting started may seem intimidating at first, but as you follow along and set up the necessary files, you’ll quickly begin to see the benefits of using these tools. By implementing these automated security scans, you’ll save time and effort while maintaining greater confidence in the security of your AWS resources. Overall, using Terraform and Prowler is an excellent investment in the security and efficiency of your cloud infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KmKvvtLsm7E3pQyVKrm8g.jpeg" /></figure><p>“If you’re a system administrator, security professional, or cloud engineer looking for a reliable and efficient way to manage the security of your AWS resources, then Terraform combined with Prowler is an excellent solution. Thank you for time you took to read this post”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b44fd3fddb9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Node with .NET 6.0 on AWS CodeBuild with Terraform]]></title>
            <link>https://medium.com/@janngoncearuc/building-node-with-net-6-0-on-aws-codebuild-with-terraform-ec89da4175e8?source=rss-6c479913df22------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec89da4175e8</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Janngoncearuc]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 11:31:28 GMT</pubDate>
            <atom:updated>2022-08-05T11:31:28.843Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Node with .NET 6.0 on AWS CodeBuild with Terraform</h3><p>Few days ago I was trying to create a build for a test application that will use .NET 6.0 on AWS using CodeBuild, but after few hours of investigation I managed to end my “trip” of research and I started to create my own Docker image, but still part of me wanted to use the “easy way” and finish the build in a more easier way for my colleagues after.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*5baeYke28kr1R84D1U0UnQ.png" /></figure><p>The first part will be to setup the codeBuild module with Terraform and for that one we can use the following module posted over the Internet:</p><p><a href="https://github.com/lgallard/terraform-aws-codebuild">GitHub - lgallard/terraform-aws-codebuild: Terraform module for creating AWS CodeBuild Projects</a></p><p>I used this module to create the codeBuild project, and I needed to create within my Project folder for Terraform , the following files :</p><p><strong>main.tf</strong></p><pre>module &quot;myapp-project&quot; {</pre><pre>source = &quot;lgallard/codebuild/aws&quot;</pre><pre>name        = &quot;my-app-var&quot;<br>  description = &quot;Codebuild for deploying myapp (variables)&quot;</pre><pre># CodeBuild Source<br>  codebuild_source_version = &quot;master&quot;</pre><pre>codebuild_source_type  = &quot;GITHUB&quot;</pre><pre>codebuild_source_buildspec = &quot;codebuild.yaml&quot;<br>codebuild_source_location =&quot;<a href="https://github.com/lgallard/codebuild-example.git">https://github.com/lgallard/codebuild-example.git</a>&quot;<br>codebuild_source_git_clone_depth                        = 0<br>codebuild_source_git_submodules_config_fetch_submodules = true</pre><pre># Environment<br>  environment_compute_type    = &quot;BUILD_GENERAL1_SMALL&quot;<br>  environment_image           = &quot;aws/codebuild/standard:5.0&quot;<br>  environment_type            = &quot;LINUX_CONTAINER&quot;<br>  environment_privileged_mode = true</pre><pre># Environment variables<br>  environment_variables = [<br>    {<br>      name  = &quot;ENV&quot;<br>      value = <em>local</em>.env_id<br>    },<br>    {<br>      name  = &quot;AWS_DEFAULT_REGION&quot;<br>      value = &quot;us-east-1&quot;<br>    },<br>  ]</pre><pre># Artifacts<br>  artifacts_location  = aws_s3_bucket.myapp-project.bucket<br>  artifacts_type      = &quot;S3&quot;<br>  artifacts_path      = &quot;/&quot;<br>  artifacts_packaging = &quot;ZIP&quot;</pre><pre># Cache<br>  cache_type     = &quot;S3&quot;<br>  cache_location = aws_s3_bucket.myapp-project.bucket</pre><pre># Logs<br>  s3_logs_status   = &quot;ENABLED&quot;<br>  s3_logs_location = &quot;${aws_s3_bucket.myapp-project.id}/build-var-log&quot;</pre><pre># Tags<br>  tags = vars.tags</pre><pre>}</pre><p><strong>variables.tf</strong></p><pre><em>variable </em>&quot;tags&quot; {<br>  description = &quot;A mapping of tags to assign to the resource.&quot;<br>  type        = map(string)<br>  default     = {<br>&quot;Environment&quot; = &quot;Testing&quot;<br>&quot;Owner&quot;       = &quot;IaC user&quot;</pre><pre>}<br>}</pre><p><strong>backend.tf </strong>: In this file we will store the state of Terraform</p><p><strong>provider.tf</strong>: We will specify the provider that we use in Terraform, by default we orient on AWS in this article as codeBuild it’s AWS Service.</p><p>Further going, we need to create<strong> codeBuild.yaml</strong> file with definition that we need for triggering the right build with .NET6.</p><p>I managed to make it work in the end by creating the following <strong>codeBuild.yaml</strong> file:</p><pre>version: 0.2<br>env:<br>phases:<br>  install:<br>    commands:<br>      - /usr/local/bin/dotnet-install.sh --channel LTS<br>  pre_build:<br>    commands:<br>      - cd src/dotnet/<br>  build:<br>    commands:<br>      - dotnet build HelloWorld.csproj</pre><p>This <strong>buildSpec</strong> worked for me using the aws/codebuild/standard:5.0 image that we use in Terraform while we create our resource in AWS.</p><p>Once the codeBuild project was created within AWS and job was triggered, then we can see the following output that will be generated by CodeBuild:</p><blockquote>Welcome to .NET 6.0!</blockquote><blockquote>— — — — — — — — — — —</blockquote><blockquote>SDK Version: 6.0.302 46 47Telemetry</blockquote><h3>Next Steps</h3><p>This article was put together hastily in order to address the lack of information on this subject.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec89da4175e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>