<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Asmita Bharti on Medium]]></title>
        <description><![CDATA[Stories by Asmita Bharti on Medium]]></description>
        <link>https://medium.com/@asmitabharti9176?source=rss-ded1d375698d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*FixC3u9kFsidXxoC</url>
            <title>Stories by Asmita Bharti on Medium</title>
            <link>https://medium.com/@asmitabharti9176?source=rss-ded1d375698d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:39:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@asmitabharti9176/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chaos Engineering for AWS Resources: A Custom Approach]]></title>
            <link>https://medium.com/expedia-group-tech/chaos-engineering-for-aws-resources-a-custom-approach-2dbd604d5df8?source=rss-ded1d375698d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dbd604d5df8</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Asmita Bharti]]></dc:creator>
            <pubDate>Tue, 26 Mar 2024 13:02:38 GMT</pubDate>
            <atom:updated>2024-07-31T08:08:47.017Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/expedia-group-tech/engineering/home">EXPEDIA GROUP TECHNOLOGY — ENGINEERING</a></h4><h4><em>Building Resilience Through Controlled Chaos and Automated Solutions</em></h4><figure><img alt="A picture capturing an individual observing a town beside a river bustling with boating activities." src="https://cdn-images-1.medium.com/max/1024/0*WakcYSGJGFF5L6so" /><figcaption>Photo by Héctor Martínez on Unsplash</figcaption></figure><p><strong><em>Introduction</em></strong><em><br>In the dynamic world of digital resources, ensuring the resilience of our systems is paramount. This document outlines our comprehensive chaos strategy, which aims to validate, fortify, and test the stability of AWS resources. Our strategy can be implemented for AWS resources like EC2, RDS, EKS, etc. We leverage </em><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/documents.html"><em>AWS Systems Manager (SSM) documents</em></a><em> and custom scripts to enhance our systems’ ability to withstand disruptions and ensure uninterrupted data services. It helps show how the system responds when critical components of its production service infrastructure are taken down. Deliberately inducing such a failure would reveal any vulnerabilities within their systems, thereby directing us towards automated solutions adept at gracefully managing similar failures in the future.</em></p><h3>Why chaos custom solution? 💡</h3><ul><li><strong>Flexible Testing:</strong> Other tools like Chaos-mesh and Chaos Toolkit were restrictive, often used with specific test cases, and suited only for specific setups like Kubernetes clusters.</li><li><strong>Customize Chaos</strong>: Custom scripts allow us to design chaos testing solutions tailored specifically to our resource needs.</li><li><strong>Adaptability</strong>: We can customize and control chaos tests to match our unique needs, making our testing more effective and adaptable to our different resources.</li></ul><figure><img alt="An image of tangled threads that are being untangled by a group of people" src="https://cdn-images-1.medium.com/max/1000/0*ik8DFFdcf1yZM2F8.jpg" /><figcaption>Image Source: Dev Community</figcaption></figure><h3>High-level diagram 🌐:</h3><figure><img alt="An image explaining the high level diagram of chao starting with jenkins to AWS System Manager, to Chaos Run Command triggering chaos." src="https://cdn-images-1.medium.com/max/1024/1*hiy_KGHXITZFCXH9HSKA5w.png" /><figcaption>High-level Diagram for chaos</figcaption></figure><p>Our chaos creation strategy operates as follows:</p><ol><li><strong>Spinnaker</strong>: Our Workflow starts from the <a href="https://spinnaker.io/">spinnaker</a>, where we can specify the choice of chaos and include other relevant parameters.</li><li><strong>Jenkins Job</strong>: A Jenkins job specifically triggers chaos experiments on our resources. This job utilizes parameters to customize the chaos tests, such as the type of chaos (e.g., Disk Stress, CPU Stress, Index Deletion), instance-filtering criteria (based on tags, instance IDs, etc.), the number of instances to target, and other relevant settings.</li><li><strong>AWS Systems Manager (SSM) Document</strong>: SSM documents that define the chaos experiments for resources (e.g., chaos for Elasticsearch, MongoDB, etc.). These documents will encapsulate the specific chaos scenarios, ensuring consistency and repeatability.</li><li><strong>Chaos Execution</strong>: The Jenkins job triggers the relevant SSM document using AWS Systems Manager’s Run Command. This initiates chaos on the targeted resource.</li><li><strong>Monitoring with Datadog</strong>: As the chaos unfolds, we can monitor the system’s performance and behavior using Datadog. This real-time monitoring allows us to gather crucial metrics and insights into how our resources and applications respond to chaotic conditions.</li></ol><figure><img alt="A meme related to having the first chaos experiment in production" src="https://cdn-images-1.medium.com/proxy/0*E2aQ7KltilFNVZE6.png" /><figcaption>Image Source: Gremlin</figcaption></figure><h3>Elasticsearch database chaos ⚡:</h3><figure><img alt="An image of elastic bands added to resemble to elastic database" src="https://cdn-images-1.medium.com/max/512/0*qZEELxkvwc4KEDPV.jpeg" /><figcaption>Image Source: Skedler</figcaption></figure><p>We have implemented our custom solution to create chaos on EC2-based databases. One such database is Elasticsearch.</p><ul><li><strong>SSM Document:</strong> In System Manager, an SSM document for creating chaos needs to be crafted for Elasticsearch. This custom script facilitates the execution of chaos scenarios based on specified actions. It utilizes <em>aws:runshellscript</em> to run various ssh commands and generate chaos. It empowers administrators to apply controlled disruptions and test the resilience of their Elasticsearch environment, ensuring its robustness under adverse conditions.</li></ul><pre># This code is licensed under the MIT License.<br># Copyright 2023 Expedia®, Inc.<br># SPDX-License-Identifier: Apache-2.0<br>{<br>  &quot;schemaVersion&quot;: &quot;2.2&quot;,<br>  &quot;parameters&quot;: {<br>    &quot;action&quot;: {<br>      &quot;type&quot;: &quot;String&quot;,<br>      &quot;allowedValues&quot;: [<br>        &quot;DiskStress&quot;,<br>        &quot;CPUStress&quot;,<br>        &quot;IndexDeletion&quot;,<br>        &quot;NodeTermination&quot;,<br>        &quot;ESserviceStart&quot;,<br>        &quot;ESserviceStop&quot;,<br>        &quot;Latency&quot;,<br>        &quot;NetworkBlock&quot;<br>      ]<br>    },<br>    &quot;Index&quot;: {<br>      &quot;type&quot;: &quot;String&quot;,<br>      &quot;default&quot;: &quot;123&quot;,<br>      &quot;description&quot;: &quot;(Optional)&quot;<br>    },<br>    &quot;InstanceIP&quot;: {<br>      &quot;type&quot;: &quot;String&quot;,<br>      &quot;default&quot;: &quot;IP&quot;,<br>      &quot;description&quot;: &quot;(Optional)&quot;<br>    },<br>    &quot;Percent&quot;: {<br>      &quot;type&quot;: &quot;String&quot;,<br>      &quot;default&quot;: &quot;80&quot;,<br>      &quot;description&quot;: &quot;(Optional)&quot;<br>    },<br>    &quot;Latency&quot;: {<br>      &quot;type&quot;: &quot;String&quot;,<br>      &quot;default&quot;: &quot;1000&quot;,<br>      &quot;description&quot;: &quot;(Optional)&quot;<br>    },<br>    &quot;Duration&quot;: {<br>      &quot;type&quot;: &quot;String&quot;,<br>      &quot;default&quot;: &quot;120&quot;,<br>      &quot;description&quot;: &quot;(Optional)&quot;<br>    }<br>  },<br>  &quot;mainSteps&quot;: [<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;DiskStress&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;DiskStress&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;sudo amazon-linux-extras install epel -y &amp;&gt;&gt; /dev/null&quot;,<br>          &quot;sudo yum install stress-ng -y &amp;&gt;&gt; /dev/null&quot;,<br>          &quot;sudo stress-ng --hdd 8 --hdd-bytes {{Percent}}% --timeout {{Duration}}s&quot;,<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;echo DiskStress executed Successfully on Instance-ID $InstanceID for {{Duration}}s &quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;CPUStress&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;CPUStress&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;sudo amazon-linux-extras install epel -y &amp;&gt;&gt; /dev/null  &quot;,<br>          &quot;sudo yum install stress-ng -y &amp;&gt;&gt; /dev/null &quot;,<br>          &quot;sudo stress-ng -c 0 -l {{Percent}} --timeout {{Duration}}s&quot;,<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;echo CPUStress executed Successfully on Instance-ID $InstanceID for {{Duration}}s&quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;IndexDeletion&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;IndexDeletion&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;sudo curl -k --cacert /elasticsearch/config/ssl/{{InstanceIP}}.crt -XDELETE -u $user:$password https://{{InstanceIP}}:9200/{{Index}}&quot;,<br>          &quot;output=$(sudo curl -k --cacert /elasticsearch/config/ssl/{{InstanceIP}}.crt -u $user:$password https://{{InstanceIP}}:9200/_cat/indices/ | grep -i &#39;{{Index}}&#39;)&quot;,<br>          &quot;if [[ -n $output ]]; then echo &#39;{{Index}} index exists&#39;; else echo &#39;{{Index}} index deleted&#39;; fi&quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;NodeTermination&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;NodeTermination&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;echo shutdown complete of Instance-ID $InstanceID&quot;,<br>          &quot;sudo shutdown -h now&quot;,<br>          &quot;echo&quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;ESserviceStart&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;ESserviceStart&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;sudo service elasticsearch start&quot;,<br>          &quot;systemctl status elasticsearch | grep Active&quot;,<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;echo ESserviceStart operation performed on Instance-ID $InstanceID&quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;ESserviceStop&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;ESserviceStop&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;sudo service elasticsearch stop&quot;,<br>          &quot;systemctl status elasticsearch | grep Active&quot;,<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;echo ESserviceStop operation performed on Instance-ID $InstanceID&quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;Latency&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;Latency&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;InstanceIp=$(wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4)&quot;,<br>          &quot;sudo yum install iproute-tc -y &amp;&gt;&gt; /dev/null&quot;,<br>          &quot;sudo tc qdisc add dev eth0 root netem delay {{Latency}}ms&quot;,<br>          &quot;echo Latency of {{Latency}}ms execute on Instance-ID $InstanceID&quot;,<br>          &quot;tc -s qdisc | grep netem | cut -d &#39; &#39; -f 11,12 &quot;,<br>          &quot;sudo curl -k --cacert /elasticsearch/config/ssl/$InstanceIp.crt -u $user:$password https://$InstanceIp:9200/_cat/indices/*?v=true&amp;s=index&quot;,<br>          &quot;sleep {{Duration}}&quot;,<br>          &quot;sudo tc qdisc del dev eth0 root netem&quot;<br>        ]<br>      }<br>    },<br>    {<br>      &quot;action&quot;: &quot;aws:runShellScript&quot;,<br>      &quot;name&quot;: &quot;NetworkBlock&quot;,<br>      &quot;precondition&quot;: {<br>        &quot;StringEquals&quot;: [<br>          &quot;{{ action }}&quot;,<br>          &quot;NetworkBlock&quot;<br>        ]<br>      },<br>      &quot;inputs&quot;: {<br>        &quot;runCommand&quot;: [<br>          &quot;InstanceID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)&quot;,<br>          &quot;InstanceIp=$(wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4)&quot;,<br>          &quot;echo $InstanceIp&quot;,<br>          &quot;sudo iptables -A INPUT -s $InstanceIp -j DROP&quot;,<br>          &quot;sudo iptables -L INPUT&quot;,<br>          &quot;echo NetworkBlock executed on Instance-ID $InstanceID for {{Duration}}s&quot;,<br>          &quot;sleep {{Duration}}&quot;,<br>          &quot;sudo iptables -D INPUT -s $InstanceIp -j DROP&quot;<br>        ]<br>      }<br>    }<br>  ]<br>}</pre><figure><img alt="An Image explaining the workflow of Chaos" src="https://cdn-images-1.medium.com/max/1024/1*OFJkKA-7vFbnwzq5teuvxQ.png" /><figcaption>Workflow of Chaos in Elasticsearch</figcaption></figure><ul><li><strong>Jenkins Job:</strong> We have created a Jenkins job to trigger chaos in EC2 instances.</li></ul><h4><strong>Jenkins job (elasticsearch-chaos-trigger) parameters</strong>:</h4><ol><li><strong>Action: </strong>Disk Stress, CPU Stress, Index Deletion, Node Termination, Elasticsearch Start/Stop, Latency and NetworkBlock.</li><li><strong>Tag Key, Tag Value: </strong>Filtering instances based on cluster name, name, instance ID.</li><li><strong>Count: </strong>Specifies the number of instances to be targeted.</li><li><strong>Index: </strong>Name of the Elasticsearch index to be deleted.</li><li><strong>Percent: </strong>This defines how much percentage of stress should be executed in case of CPU Stress and Disk Stress.</li><li><strong>Latency:</strong> Latency in milliseconds for requests.</li><li><strong>Duration: </strong>For how long an action should be performed.</li></ol><h4>Jenkins job build step (Execute Shell):</h4><pre># This code is licensed under the MIT License.<br># Copyright 2023 Expedia®, Inc.<br># SPDX-License-Identifier: Apache-2.0<br><br>#!/bin/bash<br> if [ &quot;$TAG_KEY&quot; == &quot;ClusterName&quot; ] || [ &quot;$TAG_KEY&quot; == &quot;Name&quot; ]; then<br>    TAG_KEY=tag:$TAG_KEY<br> fi<br># Use the AWS CLI to describe instances with the specified tag and limit to no.instances<br>instance_ids=$(aws ec2 describe-instances \<br>  --region &quot;us-west-2&quot; \<br>  --filters &quot;Name=instance-state-name,Values=running&quot; &quot;Name=$TAG_KEY,Values=$TAG_VALUE&quot; \<br>  --query &quot;Reservations[].Instances[].[InstanceId, Tags[?Key==&#39;Name&#39;].Value | [0]]&quot; \<br>  --output text |head -n $Count)<br><br># Check if the instance_ids variable is empty or contains only whitespace<br>if [[ -z &quot;$instance_ids&quot; ]]; then<br>  echo &quot;No instances found with the specified tag.&quot;<br>  exit 1<br>fi<br><br># Split the instance IDs into an array<br>IFS=$&#39;\n&#39; read -rd &#39;&#39; -a instance_id_array &lt;&lt;&lt; &quot;$instance_ids&quot;<br><br># Loop through the instance IDs<br>for instance_id in &quot;${instance_id_array[@]}&quot;; do<br>  echo &quot;instance ID: $instance_id&quot;<br>done<br> <br> for element in &quot;${instance_id_array[@]}&quot;; do<br>    # Remove text after the first space<br>    modifiedElement=$(echo &quot;$element&quot; | awk &#39;{print $1}&#39;)<br> <br>    modifiedArray+=(&quot;$modifiedElement&quot;)<br>done<br>modifiedElementsCSV=$(IFS=&#39;,&#39;; echo &quot;${modifiedArray[*]}&quot;)<br><br># Display the modified elements separated by commas<br>echo &quot;#########################Performing $action action for InstanceIDs: $modifiedElementsCSV ########################################&quot;<br>echo $action<br><br>InstanceIP=$(aws ec2 describe-instances \<br>  --region &quot;us-west-2&quot; \<br>  --instance-ids &quot;${modifiedArray[0]}&quot; \<br>  --query &#39;Reservations[0].Instances[0].PrivateIpAddress&#39; \<br>  --output text)<br><br><br>sh_command_id=$(aws ssm send-command \<br>    --targets Key=instanceids,Values=[&quot;$modifiedElementsCSV&quot;]\<br>    --document-name &quot;Chaos_ES&quot; \<br>    --region &quot;us-west-2&quot; \<br>    --comment &quot;run shell script on Linux Instances&quot; \<br>    --parameters &quot;{\&quot;action\&quot;:[\&quot;$action\&quot;],\&quot;InstanceIP\&quot;:[\&quot;$InstanceIP\&quot;],\&quot;Index\&quot;:[\&quot;$Index\&quot;],\&quot;Percent\&quot;:[\&quot;$Percent\&quot;],\&quot;Latency\&quot;:[\&quot;$Latency\&quot;],\&quot;Duration\&quot;:[\&quot;$Duration\&quot;]}&quot;\<br>    --output text \<br>    --query &quot;Command.CommandId&quot;) <br><br>echo &quot;CommandId: $sh_command_id&quot;<br><br># Define the desired status to check for (e.g., &quot;Success&quot;)<br>desired_status=&quot;Success&quot;<br><br># Initialize a variable to hold the current status<br>current_status=&quot;InProgress&quot;<br><br>while [ &quot;$current_status&quot; != &quot;$desired_status&quot; ]; do<br>    # Get the status of the SSM Run Command<br>    status_output=$(aws ssm list-command-invocations \<br>        --command-id &quot;$sh_command_id&quot; \<br>        --region us-west-2 \<br>        --query &quot;CommandInvocations[0].Status&quot; \<br>        --output text)<br><br>    # Check if the current status matches the desired status<br>    if [ &quot;$status_output&quot; = &quot;$desired_status&quot; ]; then<br>         echo &quot;Current status: $status_output&quot;<br>         echo &quot;###############################CommandID $sh_command_id for action $action is Success ###########################&quot;<br>         break<br>    elif [ &quot;$status_output&quot; = &quot;Failed&quot; ]; then<br>         echo &quot;Current status: $status_output&quot;<br>         echo &quot;###############################CommandID $sh_command_id for action $action is Failed ###########################&quot;<br>         break<br>    else<br>        echo &quot;###############################CommandID $sh_command_id for action $action is InProgress ###########################&quot;<br>        sleep 30<br>    fi<br><br>done<br><br>    <br><br>sh -c &quot;aws ssm list-command-invocations \<br>    --command-id \&quot;$sh_command_id\&quot; \<br>    --region us-west-2 \<br>    --details \<br>    --query &#39;CommandInvocations[].CommandPlugins[].{Status:Status,Output:Output}&#39;&quot;| sed &#39;s/----------ERROR-------/ /g&#39;</pre><p>This Jenkins job triggers the chaos SSM document, which then orchestrates chaos on the specified EC2 instances running Elasticsearch.</p><ul><li><strong>Spinnaker Pipeline: </strong>We have created a Spinnaker pipeline to trigger chaos in the Elasticsearch database consequently running the Jenkins build. The pipeline has the following parameters: Key, Value, Count, IndexName, Percent, Latency, Duration.</li></ul><figure><img alt="An image depicting the Spinnaker Pipeline for chaos" src="https://cdn-images-1.medium.com/max/1024/1*Zi2mABzmV5h5ex2ZkIlkzg.png" /><figcaption>Spinnaker Pipeline for chaos</figcaption></figure><figure><img alt="An image depicting all the values that can be added as parameters of the pipeline" src="https://cdn-images-1.medium.com/max/966/1*Vr9mGf5sK06MHdTTatLTwQ.png" /><figcaption>Values that can be added as parameters of the pipeline</figcaption></figure><h3>CPU stress output:</h3><p>We implemented CPU stress chaos on instances, and the outcomes could be verified on the Jenkins console as well as by examining Datadog metrics.</p><figure><img alt="An image depicting the jenkins console where we can verify the CPU stress chaos" src="https://cdn-images-1.medium.com/max/1024/1*RVb3EhHO-WgrcKFDFsVYXw.png" /><figcaption>Verifying CPU stress chaos on the Jenkins console</figcaption></figure><h3><strong>IndexDeletion output:</strong></h3><p>We implemented IndexDeletion chaos on instances, and the outcomes can be verified either on the Jenkins console or by checking Datadog metrics.</p><figure><img alt="An image depicting the jenkins console where we can verify the Index Deletion" src="https://cdn-images-1.medium.com/max/1024/1*q4-dPCMNOB03KarUttz45g.png" /><figcaption>Verifying Index Deletion on the Jenkins Console</figcaption></figure><h3>Monitoring with Datadog: Real-time insights during chaos experiments 🔍</h3><p>We leverage Datadog’s Dashboard as a vital element in our chaos engineering strategy for <em>ensuring the resilience</em> <em>of our EC2 databases</em>. Datadog provides real-time monitoring and actionable insights during intentional chaos inductions. It offers immediate visibility into disturbances, allowing us to assess their impact instantly. Critical performance metrics such as CPU usage, memory consumption, and query response times are deeply monitored. Datadog’s proactive alerting and anomaly detection enable swift responses to irregularities. Customized dashboards provide real-time data for specific chaos experiments, enhancing clarity. The historical data analysis feature helps identify trends and patterns in database behavior.</p><figure><img alt="An image of how a Chaos related datadog dashboard looks like" src="https://cdn-images-1.medium.com/max/1024/1*-zNcYYzTE9LBbxQ77kzE0w.png" /><figcaption>Chaos Datadog Dashboard</figcaption></figure><p>Datadog’s integration empowers data-driven decision-making, efficient incident response, continuous refinement, and validation of system resilience. It is a cornerstone in our journey to create resilient, uninterrupted data services in our EC2 databases through controlled chaos.</p><h3>Advantages 🌟</h3><p>Chaos strategy provides several key advantages:</p><ul><li><strong>Validation of Alerts</strong>: By subjecting systems to controlled chaos, we validate the effectiveness of existing monitoring and alerting mechanisms.</li><li><strong>Enhanced Backup and Restore Procedures</strong>: Chaos testing reveals gaps in backup and restore processes, allowing us to refine and fortify these critical operations.</li><li><strong>Streamlined Alerting and Restoration</strong>: We can ensure that alerting and restoration workflows operate smoothly and efficiently, minimizing downtime and disruption.</li><li><strong>Reduced RTO and RPO</strong>: Through proactive chaos testing, we can reduce both Recovery Time Objective (RTO) and Recovery Point Objective (RPO) metrics, enabling quicker recovery and minimal data loss in case of failures.</li></ul><h3>Conclusion</h3><p>Chaos strategy is a good practice for maintaining robust and resilient database systems. Similar workflows can be implemented for other stand-alone EC2, and custom scripts can be made for other resources. This will strengthen the ability to provide uninterrupted data services and test the effectiveness and stability of resources.</p><blockquote><a href="https://careers.expediagroup.com/life/">Learn about life at Expedia Group</a>™</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dbd604d5df8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/chaos-engineering-for-aws-resources-a-custom-approach-2dbd604d5df8">Chaos Engineering for AWS Resources: A Custom Approach</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS S3 Cross Account Replication]]></title>
            <link>https://medium.com/@asmitabharti9176/aws-s3-cross-account-replication-c58b87c1a647?source=rss-ded1d375698d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c58b87c1a647</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-s3-bucket]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Asmita Bharti]]></dc:creator>
            <pubDate>Fri, 04 Aug 2023 18:56:51 GMT</pubDate>
            <atom:updated>2023-10-20T05:33:19.180Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*Cnnov5AXSep67Blg.png" /><figcaption>Source Image Cloudify</figcaption></figure><blockquote><strong>Introduction: </strong>AWS S3 Cross-Account Replication allows you to automatically replicate objects from a source bucket in one AWS account to a destination bucket in another AWS account. This ensures data redundancy and fault tolerance across multiple AWS accounts. In this guide, we will walk you through the steps to set up cross-account replication between two S3 buckets.</blockquote><h3>Prerequisites:</h3><p><strong>Before proceeding with the setup, ensure the following prerequisites are met:</strong></p><ol><li>One <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html">S3 bucket</a> in the Source Account located in the source region.</li></ol><p>2. One S3 bucket in the Destination Account located in the destination region.</p><p>3. Versioning must be enabled for both the source and destination buckets.</p><p>4. Default encryption should be enabled for both the source and destination buckets.</p><p><strong>Source Account :</strong></p><ol><li>Go to the S3 console, select the source bucket, and navigate to the “Management” tab.</li><li>Click on “Replication rules” and then “Create replication rule.”1) Enter a meaningful name for the replication rule.</li></ol><p>3. For the destination, choose “Specify a bucket in another account” and provide the destination bucket name and the Destination Account ID.</p><p>4. <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html">IAM role</a> - Choose from existing IAM roles(It will automatically create after replication rules creation)</p><p>5. Save the replication rule to complete the setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*53QvFTB1H0PPR8NC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/0*15-EEHOIaRnYziJ4" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFF_nHXJb-myOh11FqOSsA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yCpne5ZcAgHMi-95dW2SdQ.png" /></figure><p><strong>Destination Account :</strong></p><ol><li>Go to the S3 console, select the destination bucket, and navigate to the “Permissions” tab.</li><li>Click on “Edit bucket policy” and insert the following bucket policy. Make sure to replace “SourceAccountID” with the actual Source Account ID and “DestinationBucketName” with the appropriate destination bucket name.After pasting the policy, save the changes to apply the configuration.</li></ol><pre><br>{<br>    &quot;Version&quot;: &quot;2008-10-17&quot;,<br>    &quot;Id&quot;: &quot;S3-Console-Replication-Policy&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;S3ReplicationPolicyStmt1&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::SourceAccountID:root&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;s3:GetBucketVersioning&quot;,<br>                &quot;s3:PutBucketVersioning&quot;,<br>                &quot;s3:ReplicateObject&quot;,<br>                &quot;s3:ReplicateDelete&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::DestinationBucketName &quot;,<br>                &quot;arn:aws:s3::: DestinationBucketName /*&quot;<br>            ]<br>        }<br>    ]<br>}<br><br></pre><p>Final Steps:</p><ol><li>In the Destination Account’s S3 console, go to the “Management” tab, then “Replication rules.”</li><li>Select the appropriate replication rule and click on “Actions.”</li><li>Choose “Receive Replicated Objects” and enter the Source Account ID to establish the replication link.</li><li>Close the configuration window to complete the process.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KSXDz9sFOYwve6a9" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2K_0nD8mZB9tpH9bF6gQxA.png" /></figure><p>Validation:</p><ol><li>Upload files to the source bucket.</li><li>Check the destination bucket to ensure that the files are successfully replicated from the Source Account to the Destination Account.</li></ol><p><strong><em>Conclusion:</em></strong></p><p><em>By following this guide, you have successfully set up AWS S3 Cross-Account Replication between two AWS accounts. This configuration ensures that objects in the source bucket are automatically replicated to the destination bucket, providing data redundancy and protection against data loss.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c58b87c1a647" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Windows EC2 Instances in AWS Autoscaling Group with Datadog and Creating Dashboard]]></title>
            <link>https://medium.com/@asmitabharti9176/monitoring-windows-ec2-instances-in-aws-autoscaling-group-with-datadog-and-creating-dashboard-611f00ac72d4?source=rss-ded1d375698d------2</link>
            <guid isPermaLink="false">https://medium.com/p/611f00ac72d4</guid>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[datadog-agent]]></category>
            <category><![CDATA[datadog]]></category>
            <dc:creator><![CDATA[Asmita Bharti]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 19:32:16 GMT</pubDate>
            <atom:updated>2023-07-24T19:41:33.563Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qo7Ndzwx0HKqoZXSUi5_0A.png" /></figure><blockquote><strong>Introduction:</strong><br>In this article, we will explore the process of installing the <strong>Datadog agent on Windows EC2 instances</strong> that are part of an Autoscaling group in AWS. We will also delve into the challenges faced during the setup and configuration of Datadog dashboards to effectively monitor and visualize key metrics of the Windows Bastion and its associated load balancer, such as disk usage, CPU utilization, and the number of healthy hosts.</blockquote><p><strong>Monitoring on Windows EC2 Instances:</strong><br>To begin monitoring our Windows EC2 instances using Datadog, we first need to install the <a href="https://docs.datadoghq.com/agent/basic_agent_usage/windows/?tab=gui">Datadog agent</a> on each instance. The Datadog agent collects system-level metrics, custom application metrics, and traces to provide a comprehensive view of the system’s health and performance.</p><p>To ensure seamless installation and monitoring, we have opted for a tag-based datadog agent installation approach. By tagging our Windows EC2 instances within the Autoscaling group, we can easily associate them with the appropriate Datadog dashboard even during instance replacements.</p><p>One of the main challenges faced during the setup was ensuring that the Datadog dashboard remains linked to the Windows Bastion instance even when it is replaced due to scaling actions or instance failures. By leveraging tags, we could uniquely identify each instance, and even if a new instance is launched or an existing one terminated, the dashboard would automatically update to include the new instance and remove the terminated one.</p><p>Datadog Agent Installation Process:</p><ol><li>Download Datadog agent for windows from below link:<br>https://s3.amazonaws.com/ddagent-windows-stable/datadog-agent-7-latest.amd64.msi</li></ol><pre>https://s3.amazonaws.com/ddagent-windows-stable/datadog-agent-7-latest.amd64.msi</pre><p>2.Create a powershell script to install datadog agent with tag such as account name or autoscaling name and save it with .ps1 extension:</p><pre>Start-Process -Wait msiexec -ArgumentList /i, C:\\winbastion\\datadog-agent-7-latest.amd64.msi, /qn, /passive, APIKEY=&quot;Your_API_Key&quot;, TAGS=&quot;Key:Value&quot;</pre><p>Alternatively both download and install command can be put:</p><pre>Invoke-WebRequest -Uri &#39;https://s3.amazonaws.com/ddagent-windows-stable/datadog-agent-7-latest.amd64.msi&#39; -OutFile &#39;C:\winbastion\datadog-agent-7-latest.amd64.msi&#39;<br>Start-Process -Wait msiexec -ArgumentList /i, C:\\winbastion\\datadog-agent-7-latest.amd64.msi, /qn, /passive, APIKEY=&quot;Your_API_Key&quot;, TAGS=&quot;Key:Value&quot;</pre><p>3. Put both Installation package as well as Powershell script in s3 bucket location and ensure that ec2 instance should have access to this particular bucket via iam role.</p><p>4.Make changes in userdata to add powershell commands to copy content from s3 bucket as well as creation of task scheduler that will trigger installation script at startup.</p><pre>&quot;&lt;powershell&gt;\n&quot;,<br>&quot;$S3BucketName = &quot;S3Bucket&quot;\n&quot;,<br>&quot;Read-S3Object -BucketName $S3BucketName -KeyPrefix winbastion -Folder C:\\winbastion\n&quot;,<br>&quot;$ScriptPath = &#39;C:\\winbastion\\Datadog_Install.ps1&#39; \n &quot;,<br>&quot;$Trigger= New-ScheduledTaskTrigger -AtStartup \n&quot;,<br>&quot;$Action= New-ScheduledTaskAction -Execute \&quot;PowerShell.exe\&quot; -Argument \&quot;-executionpolicy bypass -noprofile -file $ScriptPath\&quot; \n&quot;,<br>&quot;Register-ScheduledTask -TaskName &#39;DatadogInstall&#39; -Trigger $Trigger -User &#39;NT AUTHORITY\\SYSTEM&#39; -Action $Action -RunLevel Highest \n&quot;,<br>&quot;Restart-Computer -Force\n&quot;,<br>&quot;&lt;/powershell&gt;\n&quot;</pre><p><strong>Challenge:</strong><br>Userdata was having other instructions as well that needed system restart because of which datadog agent was not getting started and if we put datadog restart command in userdata itself , it was preventing system to restart and making fail other commands<br><strong>Solution: <br></strong>Created a powershell script to install and download datadog agent and put it in s3 bucket and put commands to create task scheduler to run this script at startup. This approach ensured that the agent starts collecting metrics as soon as the instance boots up, providing real-time monitoring and preventing any delays in data acquisition.</p><p>Creating a Datadog Dashboard for Windows Bastion and Load Balancer Monitoring:<br>With the Datadog agent successfully installed and collecting data from our Windows EC2 instances, we proceeded to create a comprehensive dashboard to visualize the key metrics.</p><p>Key Metrics for Monitoring Windows Bastion and Load Balancer:<br>- Disk Usage: Monitoring disk usage is crucial to prevent storage-related issues and ensure sufficient space is available for smooth operations.<br>- CPU Utilization: Keeping track of CPU utilization helps identify performance bottlenecks and potential resource constraints.<br>- Number of Healthy Hosts: This metric is vital for load balancer monitoring, ensuring that the load balancer is distributing traffic to healthy instances.</p><p><strong>Datadog Dashboard Creation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFiqbmtYAujyG_TuBh5Ipg.png" /></figure><p>Using the Datadog web interface, we crafted a custom dashboard tailored to our specific monitoring needs. We added widgets to display metrics like disk usage, CPU utilization, and the number of healthy hosts. Additionally, we set up alerting thresholds to receive notifications when any metric breaches predefined limits, allowing for proactive issue resolution.</p><p><strong>Conclusion:</strong><br>Monitoring Windows EC2 instances within an Autoscaling group is crucial for maintaining optimal performance and addressing potential issues promptly. By leveraging the Datadog agent and dashboard creation capabilities, we can effectively monitor and visualize critical metrics of our Windows Bastion and associated load balancer. Overcoming challenges related to instance replacement and agent startup ensures a seamless monitoring experience, enabling us to proactively respond to any deviations from expected behaviour. With Datadog, we gain valuable insights that empower us to make data-driven decisions and maintain a reliable and efficient infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=611f00ac72d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>