<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dipto Chakrabarty on Medium]]></title>
        <description><![CDATA[Stories by Dipto Chakrabarty on Medium]]></description>
        <link>https://medium.com/@diptochakrabarty?source=rss-6a7d00f09c48------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9rEs3zqeyHHGk_k6qkeX2A.jpeg</url>
            <title>Stories by Dipto Chakrabarty on Medium</title>
            <link>https://medium.com/@diptochakrabarty?source=rss-6a7d00f09c48------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:01:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@diptochakrabarty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Journey Working with Scylla — Part 1]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://diptochakrabarty.medium.com/my-journey-working-with-scylla-part-1-4170c4db3c28?source=rss-6a7d00f09c48------2"><img src="https://cdn-images-1.medium.com/max/600/1*8UcTuAa_fy9akmWLvXpsWA.jpeg" width="600"></a></p><p class="medium-feed-snippet">I started working with Scylla and this is my documentation on how that is going.</p><p class="medium-feed-link"><a href="https://diptochakrabarty.medium.com/my-journey-working-with-scylla-part-1-4170c4db3c28?source=rss-6a7d00f09c48------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://diptochakrabarty.medium.com/my-journey-working-with-scylla-part-1-4170c4db3c28?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/4170c4db3c28</guid>
            <category><![CDATA[scylladb]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Sun, 24 May 2026 04:51:00 GMT</pubDate>
            <atom:updated>2026-05-24T04:51:00.439Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Using S3 to save 80% costs of moving 20 Million worth of data to DynamoDB]]></title>
            <link>https://diptochakrabarty.medium.com/using-s3-to-save-80-costs-of-moving-20-million-worth-of-data-to-dynamodb-ee0bda3ff44c?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee0bda3ff44c</guid>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Tue, 22 Apr 2025 23:06:08 GMT</pubDate>
            <atom:updated>2025-04-22T23:06:08.524Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cloud Computing | Databases</h4><h4>How we saved 80% of our costs by loading data from S3 to dyanmodb</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*glnuceiJ-MWxMqzz" /><figcaption>Photo by <a href="https://unsplash.com/@ikukevk?utm_source=medium&amp;utm_medium=referral">Kevin Ku</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>The problem was simple , we wanted to perform a POC in dynamoDB which needed us to move our entire data of 20 Million records.</p><p>Currently the data was stored in a mysql database and we needed to figure out a way to move this to dynamodb.</p><h3>The problem</h3><p>We had a limited budget which we did not want to overshoot just for performing a proof of concept.</p><p>DynamoDB charges for each write request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mvuLcJGzFbaSJ9EHKt_TuA.png" /></figure><p>Now due to dynamoDB’s restrictive schema designs we had to duplicate our data to 40 Million records so our total cost in worst case would come to be</p><p>40 x 0.78 = 31 dollars.</p><p>We did not want to spend that much money just to get our data there.</p><h3>The solution</h3><p>DynamoDB has a solution to migrate data from S3 which we found really cheap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dhpLfXtEBeGS845bSAVa_A.png" /></figure><p>All we had to ensure was data was in the same format expected by dynamoDB and that&#39;s it.</p><p>Our data size was around 12 GB so the total cost came to 12 x 0.15 = 6 dollars</p><p>That is a <strong>whopping 80% reduction</strong> in cost.</p><p>And since data storage in S3 is a lot lot cheaper this was the right solution for us.</p><h3>The process</h3><p>Move your data to S3 as json , a sample data looks like this</p><pre>{<br>  &quot;Item&quot;: {<br>    &quot;pk&quot;: {&quot;S&quot;: &quot;r#874&quot;},<br>    &quot;sk&quot;: {&quot;S&quot;: &quot;630971678365982720&quot;},<br>    &quot;source_user_id&quot;: {&quot;N&quot;: &quot;174305318&quot;},<br>    &quot;source_screen_name&quot;: {&quot;S&quot;: &quot;DLGreenfield&quot;},<br>    &quot;source_description&quot;: {&quot;S&quot;: &quot;Education strategist @ucls @GrinnellCollege&quot;},<br>    &quot;target_user_id&quot;: {&quot;N&quot;: &quot;874&quot;},<br>    &quot;target_screen_name&quot;: {&quot;S&quot;: &quot;waynesutton&quot;},<br>    &quot;target_description&quot;: {&quot;S&quot;: &quot;Cofounder @changecatalysts @buildupvc&quot;},<br>    &quot;created_at&quot;: {&quot;S&quot;: &quot;2015-08-11T05:19:05&quot;},<br>    &quot;text&quot;: {&quot;S&quot;: &quot;Join @square @blackfounders speaking&quot;},<br>    &quot;hashtags&quot;: {&quot;S&quot;: &quot;techinclusion15&quot;},<br>    &quot;combined_score&quot;: {&quot;N&quot;: &quot;0.6931471805599453&quot;}<br>  }<br>}</pre><p>This gives a clear indication to dynamoDB the type of field , S stands for String , N for number and pk is partition key and sk is sort key as I have named them.</p><p>In dynamoDB click on import from S3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*gtOnd9inmUbJKMYq3FMCmQ.png" /></figure><p>Set your S3 URL and choose a compression if you feel compressing it will benefit you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C8eZuxmszg73-y7F8O3vZg.png" /></figure><p>Set your table name and choose the primary key and sort key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qnACKJA5hlCBJU6DX-F-Dw.png" /></figure><p>You can customize your settings or change the settings if needed but the default one was fine for us anyway.</p><p>And click next and your table creation should have started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nWc-D_fmVqlwz7QqeRetIg.png" /></figure><p>You will be able to see a new table created for you.</p><p>And this comes at a very cheap price , you are essentially compromising on financial costs to manual cost of migrating the data to S3 in the right format.</p><h3>Conclusion</h3><p>We found this the best way to migrate massive data to dynamoDB , there might be something better and if you do find it please make sure to comment it below for me to explore.</p><p>Till then if you liked the article make sure to like it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee0bda3ff44c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Be Smart , Save Money and Run DynamoDB Locally Before AWS]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://diptochakrabarty.medium.com/be-smart-save-money-and-run-dynamodb-locally-before-aws-992447c8605e?source=rss-6a7d00f09c48------2"><img src="https://cdn-images-1.medium.com/max/2600/0*7j_YqOuAMY2ziQ4h" width="2947"></a></p><p class="medium-feed-snippet">How we are saving money by first testing dynamoDB locally in our machine</p><p class="medium-feed-link"><a href="https://diptochakrabarty.medium.com/be-smart-save-money-and-run-dynamodb-locally-before-aws-992447c8605e?source=rss-6a7d00f09c48------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://diptochakrabarty.medium.com/be-smart-save-money-and-run-dynamodb-locally-before-aws-992447c8605e?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/992447c8605e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[money]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 13:21:17 GMT</pubDate>
            <atom:updated>2025-04-18T13:21:17.468Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Restoring your Mysql Database using a snapshot as fast as the Flash]]></title>
            <link>https://diptochakrabarty.medium.com/restoring-your-mysql-database-using-a-snapshot-as-fast-as-the-flash-1e884d8e1d8e?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e884d8e1d8e</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Fri, 18 Apr 2025 01:31:50 GMT</pubDate>
            <atom:updated>2025-04-18T01:31:50.443Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cloud Computing | Databases</h4><h4>This is all about how to revive your mysql database using previous volume snapshots</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XuKhfN8bYUiFIpcK" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>I was working on a project where I had to often load and unload data from a mysql database. Since I was using AWS and the project was such that I needed to have mysql for a limited few hours I would have to recreate the database from scratch and migrate the whole data.</p><p>The need for snapshots came with two clear benefits</p><ul><li>I could recreate my entire mysql database any time I wanted</li><li>I did not have to remigrate the data</li></ul><p>So lets begin with how to do this</p><h3>Initial Instance Setup</h3><p>As a first step create a new instance in AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GVfYNqdQsrAkzryafoGWtA.png" /></figure><p>Make sure the attached security group in the instance allows the mysql port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uxhY6kIwQSgLTO3dUrWbkA.png" /></figure><p>Now I had already prepared a snapshot of the EBS volume where I had previously loaded my mysql database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZauUA_pihWyT6sYnZu1Nog.png" /></figure><p>Create a new volume with this snapshot and wait till its ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3VbAysDKhHrmLml_i6V9PA.png" /></figure><p>Next attach this volume to the instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zzm6GXS8i5BFQZMAYL60WA.png" /></figure><p>Set a device name which will be accessible inside AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZSwThAqipoR7zSsrKRIjA.png" /></figure><p>Now part 1 is done, lets jump to the next step.</p><h3>Preparing the Instance</h3><p>First press the following command which shows your active mounts in the system</p><pre>df -hT</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*uFzClMyqDDfMR4hYgrDSKA.png" /></figure><p>You will notice the volume is not mounted so press the following command which displays the exisiting logical volumes in your instance.</p><pre>lsblk</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*I-CUuPlTQRX6AJc37Na5tw.png" /></figure><p>If you notice xvdd is 30GB big , this is the volume containing mysql data. Dont make the mistake of formatting this since all data will be lost.</p><p>This is because the snapshot volume comes pre baked with mysql data and we only need to mount it.</p><pre>sudo mkdir /mnt/mysql-data<br>sudo mount /dev/xvdd /mnt/mysql-data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*z7fenrn7OBcCTVpbzhsCvw.png" /></figure><p>The above two commands mounts the volume. The next step is to stop mysql and give the directory necessary permissions.</p><pre>sudo chown -R mysql:mysql /mnt/mysql-data<br>sudo chmod -R 750 /mnt/mysql-data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*pd-Zw6pAJ_MRqffMOwIoSg.png" /></figure><p>These commands run only when mysql is installed in the ubuntu machine.</p><p>Next we need to edit apparmor an internal security implementation in ubuntu AWS.</p><pre>sudo vim  /etc/apparmor.d/usr.sbin.mysqld<br><br>Add these two lines<br>/mnt/mysql-data/ r,<br>/mnt/mysql-data/** rwk,  <br><br>and restart apparmor<br>sudo systemctl restart apparmor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*oM4McRGPphz_GuHgMQARvQ.png" /></figure><h3>Mysql Configuration Changes</h3><p>The next step is make sure mysql knows which directory to use , so edit the mysql configuration file and add your directory as the data directory of mysql which is denoted by datadir.</p><pre> sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*byi-g6SzqlP6k7qUj0ngAQ.png" /></figure><p>Also you might need to add permission for the mysql error log directory</p><pre>sudo mkdir -p /var/log/mysql<br>sudo chown mysql:mysql /var/log/mysql</pre><p>The next step is the start mysql and login</p><pre>sudo systemctl daemon-reload<br>sudo systemctl start mysql<br>sudo systemctl status mysql</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*UG99I_gj2QwbPJcLzFXp_A.png" /></figure><p>Now login to mysql using your previous credentials , you don&#39;t have to create new ones and wala.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*xsY51FcWFxpURbiB8VT80w.png" /></figure><h3>Conclusion</h3><p>Snapshots make it really convenient to load mysql data and recreate mysql instances fast. This whole process can be automated using ansible if needed.</p><p>The value of snapshots also come at kubernetes since they can be directly used to create EBS volume backed mysql pods on the cluster.</p><p>Overall it is a super useful thing to make use of.</p><p>If you like the article please like and subscribe.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e884d8e1d8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy mysql simple way in Kubernetes no fluff and fast]]></title>
            <link>https://diptochakrabarty.medium.com/deploy-mysql-simple-way-in-kubernetes-no-fluff-and-fast-b6ff8e981db1?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6ff8e981db1</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Wed, 26 Mar 2025 01:10:41 GMT</pubDate>
            <atom:updated>2025-03-26T01:10:41.515Z</atom:updated>
            <content:encoded><![CDATA[<h4>K8s| Database</h4><h4>How to deploy mysql in your Kubernetes cluster</h4><p>This is a free short no fluff fast guide to deploy mysql in Kubernetes cluster locally</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X6S8rFe3SUZCf2Q7" /><figcaption>Photo by <a href="https://unsplash.com/@abject?utm_source=medium&amp;utm_medium=referral">benjamin lehman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Setting up storage class</h3><p>Using a storage class means you can provision volumes dynamically without creating a persistent volume.</p><pre>apiVersion: storage.k8s.io/v1<br>kind: StorageClass<br>metadata:<br>  name: local-mysql<br>  annotations:<br>    storageclass.kubernetes.io/is-default-class: &quot;true&quot;<br>provisioner: k8s.io/minikube-hostpath<br>reclaimPolicy: Delete<br>volumeBindingMode: Immediate</pre><p>The provisioner used is specific for minikube and annotations too.</p><p>The reclaim policy is set to delete to delete when claim is released.</p><h3>Setup a PVC</h3><p>Use a pvc for for getting the storage. Reference the storage class to use our storage class made above.</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: mysql-pvc<br>spec:<br>  storageClassName: local-mysql<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 100Mi</pre><h3>Setup a Statefulset for mysql</h3><p>We set the username and passwords as env vars but ideally it should be done using secrets.</p><pre>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: mysql<br>spec:<br>  serviceName: mysql-svc<br>  selector:<br>    matchLabels:<br>      app: mysql<br>  template:<br>    metadata:<br>      labels:<br>        app: mysql<br>    spec:<br>      containers:<br>      - name: mysql<br>        image: mysql:8.0<br>        env:<br>        - name: MYSQL_ROOT_PASSWORD<br>          value: &quot;toortoor&quot; <br>        - name: MYSQL_USER<br>          value: &quot;dipto&quot;<br>        - name: MYSQL_PASSWORD<br>          value: &quot;toortoor&quot;<br>        - name: MYSQL_DATABASE<br>          value: &quot;diptodb&quot;<br>        ports:<br>        - containerPort: 3306<br>          name: mysql<br>        volumeMounts:<br>        - name: mysql-storage<br>          mountPath: /var/lib/mysql<br>      volumes:<br>      - name: mysql-storage<br>        persistentVolumeClaim:<br>          claimName: mysql-pvc</pre><p>We reference the PVC using volumes and we mount that volume using volume Mounts.</p><h3>Setup a service</h3><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mysql-svc<br>spec:<br>  selector:<br>    app: mysql<br>  ports:<br>  - port: 3306<br>    targetPort: 3306<br></pre><p>Now apply your changes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*_HH3CcgV6pmo5VUdbAulAw.png" /><figcaption>apply the files</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*GVyyHMuxE7scrMM_tmeFuw.png" /><figcaption>check if storage class is made</figcaption></figure><p>Beware for minikube we have to enable storage class addon</p><pre>minikube addons enable default-storageclass</pre><p>New get all and see if everyting is deployed</p><pre>kubectl get all</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*uzpSvcXjcW1oexI70nOv1Q.png" /><figcaption>get all output</figcaption></figure><p>Exec into mysql pod and check root credentials</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*n3VZyAA8cHd1TpnFcbMOAA.png" /><figcaption>working with root auth</figcaption></figure><p>Now check with the user you created that is set as env vars</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*ZttY1w_uwTykgrZn9gUcjg.png" /><figcaption>new user created and loging in</figcaption></figure><p>Check is database is created or not</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*CuT4bPgaAcSqCW0rjMHKiw.png" /><figcaption>database output</figcaption></figure><h3>Conclusion</h3><p>This is a baseline of running mysql in kubernetes. For production you would ideally use secrets for credentials and usually use EBS volumes if using AWS.</p><p>I just did this while working on my project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6ff8e981db1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Learning Spark UI is like eating a good bowl of Noodles]]></title>
            <link>https://diptochakrabarty.medium.com/how-learning-spark-ui-is-like-eating-a-good-bowl-of-noodles-85b8cc617f76?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/85b8cc617f76</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Sun, 23 Mar 2025 00:57:46 GMT</pubDate>
            <atom:updated>2025-03-23T00:57:46.459Z</atom:updated>
            <content:encoded><![CDATA[<h4>Spark | ETL</h4><h4>How we are learning to use Spark UI for our ETL jobs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WXHrg6obvXPt4vwj" /><figcaption>Photo by <a href="https://unsplash.com/@davealmine?utm_source=medium&amp;utm_medium=referral">Dawid Zawiła</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you have read the title you are obviously wondering what does that mean. Well let me explain the context.</p><p>We are working on a project to process 1TB worth of data with several transformations , filtering and what not.</p><p>Along the lines first time working with Spark we are learning how to understand the UI.</p><p>Spark UI is straightforward to use but it is like a good bowl of Noodles where sometimes strands mix up together and you dont know how to split them apart or differentiate them.</p><p>Although this is not complex it is all about how we are grasping this process.</p><h3>The hierarchy of Spark</h3><p>In Spark when we submit a Spark application it is broken down into a Job , a job is broken into stages and the stages into tasks.</p><p>And what do they mean.</p><p>Lets understand with an analogy of cooking for a party.</p><h4>Job</h4><p>A job is the big level task. For example you planning to cook for the party is the job.</p><p>In spark job represents overall action you wish to achieve. These are triggered by actions like count , collect etc. So for a spark application it can have multiple jobs.</p><h4>Stage</h4><p>A job is broken down into stages. So for your dinner party you decide on three dishes which become your stages</p><ul><li>Stage 1: A salad</li><li>Stage 2: A curry</li><li>Stage3: A desert</li></ul><p>Stages are independent parts of the application that can be achieved without shuffling the data.</p><h4>Task</h4><p>Each stage is further broken down into task. So for example for preparing the salad</p><ul><li>Task1_1: Clean the vegetables</li><li>Task1_2: Cut the vegetables</li><li>Task1_3: Prepare a dressing</li></ul><p>You get the point. A task is the atomic unit of execution in spark which run on individual partitions of data.</p><p>Lets get onto monitoring the UI now.</p><h3>The Jobs Section</h3><p>Head over to the jobs section in Spark and you can see your active jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6K7l4_19XEwfDOUKjW8UOg.png" /></figure><p>Clearly you can see the second one is running heavily with multiple tasks.</p><p>Lets click on the job to get more granular details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USBSqbMYwEv7uFRXzdB_GQ.png" /></figure><p>Hmm can we see the dag view. Well click on the dag visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*sV2YXliTSMWQCVOhdZN0eA.png" /></figure><p>This dag view is a visual representation of how Spark is executing the program. In spark applications the flow of data is in a directed acyclic graph and this above image showcases this.</p><p>Lets click on the stage and we notice that it has ingested 5.8GB of data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAp4qleNnS9rg9jV8Zec8w.png" /></figure><p>On clicking on that we notice some more data about that particular stage. This is important as we get to know which part of the code is needing more time and taking more memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*Hz5YmiVolN1yT0LxZvjtlg.png" /></figure><p>To see more data click on the show additional metrics which have more options related to memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iWoWA4_OYLLNlIizDTzqhQ.png" /></figure><h3>Storage</h3><p>The storage section is as expected the amount of disk Spark is using.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z87ZRJotgRPQGHIwPaiBRQ.png" /></figure><h3>Executors</h3><p>Executors are what you can consider the chef that cooks food in the dinner party analogy , they perform the actual task.</p><p>Stages , tasks are splitting of tasks but executors are the one who perform them.</p><p>If we head over to the executors tab we can see how much pressure we are putting on each executor and are we overutilizing or underutilizing them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFxL02c9EysHkbsR_djYmQ.png" /></figure><p>We can see the memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*-kY-cb4w6a7TZ43myuBuJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*JU5DYJCLiO9Xw-oPdAUpzA.png" /></figure><p>A image of a count job</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u0FXDvwjyDwa9p8M6W4lA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hfrVKRoYJou4IkHRh8LSRg.png" /></figure><h3>How we are using it</h3><ul><li>Finding which part of our code is a bottleneck. The spark UI splits parts of the code and runs them on multiple executors and we use it to identify which part is slow or we need to optimize.</li><li>Efficient use of resources. We want to make sure we are properly utilizing the full power of our cluster or not overusing the cluster by adjusting the parameters.</li></ul><h3>Conclusion</h3><p>Spark is an interesting tool with a big learning curve. We started out with bad code and bad parameters and now we are gradually learning how to optimize our code and efficiently use the resources.</p><p>All this is possible because of the Spark UI and us learning how to navigate it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85b8cc617f76" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Bench can help your local API break its limit and become a protagonist]]></title>
            <link>https://diptochakrabarty.medium.com/apache-bench-can-help-your-local-api-break-its-limit-and-become-a-protagonist-62a6c92fabfb?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/62a6c92fabfb</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[students]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Fri, 21 Mar 2025 00:52:47 GMT</pubDate>
            <atom:updated>2025-03-21T00:52:47.437Z</atom:updated>
            <content:encoded><![CDATA[<h4>Webapp | Optimization</h4><h4>How to use Apache Bench locally to test efficiency of your web app</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ARXhWQryyDb2bQjC" /><figcaption>Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">CHUTTERSNAP</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So here we are developing an API for the cloud computing course at CMU 15–619 and we are at a dilemma how can we check how well our api is performing with new optimizations.</p><p>Every optimization we make does not contribute to the performance of the API so how do we test it.</p><p>And like a rough diamond under the soil we found and then used Apache Bench to easily do this locally.</p><p>This is a free guide on how to do this locally for your system.</p><h3>Developing the API</h3><p>I am going to write the API in golang so first lets initialize the project.</p><pre>go mod init jperf</pre><p>Next lets write two simple endpoints in fasthttp which simulates the same functionality but one is slow and other is fast</p><pre>package main<br><br>import (<br> &quot;log&quot;<br> &quot;time&quot;<br><br> &quot;github.com/valyala/fasthttp&quot;<br>)<br><br>func slowHandler(ctx *fasthttp.RequestCtx) {<br> time.Sleep(3 * time.Second)<br> ctx.SetStatusCode(fasthttp.StatusOK)<br> ctx.SetBody([]byte(&quot;Slow Response&quot;))<br>}<br><br>func fastHandler(ctx *fasthttp.RequestCtx) {<br> ctx.SetStatusCode(fasthttp.StatusOK)<br> ctx.SetBody([]byte(&quot;Fast Response&quot;))<br>}<br><br>func requestHandler(ctx *fasthttp.RequestCtx) {<br> switch string(ctx.Path()) {<br> case &quot;/slow&quot;:<br>  slowHandler(ctx)<br> case &quot;/fast&quot;:<br>  fastHandler(ctx)<br> }<br>}<br><br>func main() {<br> log.Println(&quot;Starting server at :8080...&quot;)<br> if err := fasthttp.ListenAndServe(&quot;:8080&quot;, requestHandler); err != nil {<br>  log.Fatalf(&quot;Error starting server: %v&quot;, err)<br> }<br>}</pre><p>The slow endpoint sleeps for 3 seconds and then responds while the fast one does it immediately.</p><p>Lets install the required packages</p><pre>go mod tidy</pre><p>and run the app to once test the functionalities</p><pre>go run main.go<br><br>//In different terminal<br>curl http://localhost:8080/slow<br><br>curl http://localhost:8080/fast</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*hSqgOSFXK5vzH311vUJD9A.png" /><figcaption>Clearly one is slow and other fast</figcaption></figure><p>Now lets move to the fun part.</p><h3>Setting up Apache Bench</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*fVYae7-x4EsTzH73b6jtXQ.png" /><figcaption>Apache Bench</figcaption></figure><p>If you are on windows head <a href="https://www.apachelounge.com/download/#google_vignette">over to this website</a>.</p><p>Extract the download and view the contents.</p><p>Next head to Apache24/bin and we will be working with the ab.exe</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*fzmYS4EjThr-ECTsMh6HBw.png" /><figcaption>we will be working with ab</figcaption></figure><p>Now lets test.</p><h3>Performing the Actual Tests</h3><p>Firstly run the api in a separate terminal.</p><pre>go run main.go</pre><p>Now our api is running in port 8080 we can forward requests</p><p>We will first hit the /slow endpoint</p><pre>./ab -n 50 -c 10 &quot;http://localhost:8080/slow&quot;</pre><p>Here we use two parameters</p><p>n: Total number of requests</p><p>c: The number of concurrent requests at a time</p><p>Lets look at the results</p><pre>./ab -n 50 -c 10 &quot;http://localhost:8080/slow&quot;<br>This is ApacheBench, Version 2.3 &lt;$Revision: 1923142 $&gt;<br>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/<br>Licensed to The Apache Software Foundation, http://www.apache.org/<br><br>Benchmarking localhost (be patient).....done<br><br><br>Server Software:        fasthttp<br>Server Hostname:        localhost<br>Server Port:            8080<br><br>Document Path:          /slow<br>Document Length:        13 bytes<br><br>Concurrency Level:      10<br>Time taken for tests:   20.090 seconds<br>Complete requests:      50<br>Failed requests:        0<br>Total transferred:      8350 bytes<br>HTML transferred:       650 bytes<br>Requests per second:    2.49 [#/sec] (mean)<br>Time per request:       4017.983 [ms] (mean)<br>Time per request:       401.798 [ms] (mean, across all concurrent requests)<br>Transfer rate:          0.41 [Kbytes/sec] received<br><br>Connection Times (ms)<br>              min  mean[+/-sd] median   max<br>Connect:        0    0   0.3      0       1<br>Processing:  3000 3008   5.6   3007    3017<br>Waiting:     3000 3007   5.6   3006    3017<br>Total:       3001 3009   5.6   3008    3018<br><br>Percentage of the requests served within a certain time (ms)<br>  50%   3008<br>  66%   3011<br>  75%   3014<br>  80%   3016<br>  90%   3017<br>  95%   3017<br>  98%   3018<br>  99%   3018<br> 100%   3018 (longest request)</pre><p>We completed 50 requests and took 20 seconds approximately with an average of 2.49 seconds of RPS.</p><p>Lets run it for the fast endpoint now</p><pre>./ab -n 50 -c 10 &quot;http://localhost:8080/fast&quot;</pre><p>Remember in fast the results are returned immediately and in slow we wait for 3 seconds.</p><p>Lets check the results.</p><pre>./ab -n 50 -c 10 &quot;http://localhost:8080/fast&quot;<br>This is ApacheBench, Version 2.3 &lt;$Revision: 1923142 $&gt;<br>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/<br>Licensed to The Apache Software Foundation, http://www.apache.org/<br><br>Benchmarking localhost (be patient).....done<br><br><br>Server Software:        fasthttp<br>Server Hostname:        localhost<br>Server Port:            8080<br><br>Document Path:          /fast<br>Document Length:        13 bytes<br><br>Concurrency Level:      10<br>Time taken for tests:   2.052 seconds<br>Complete requests:      50<br>Failed requests:        0<br>Total transferred:      8350 bytes<br>HTML transferred:       650 bytes<br>Requests per second:    24.37 [#/sec] (mean)<br>Time per request:       410.342 [ms] (mean)<br>Time per request:       41.034 [ms] (mean, across all concurrent requests)<br>Transfer rate:          3.97 [Kbytes/sec] received<br><br>Connection Times (ms)<br>              min  mean[+/-sd] median   max<br>Connect:        0    0   0.3      1       1<br>Processing:     1    3   1.4      3       6<br>Waiting:        0    2   1.3      2       5<br>Total:          1    4   1.5      4       6<br>ERROR: The median and mean for the initial connection time are more than twice the standard<br>       deviation apart. These results are NOT reliable.<br><br>Percentage of the requests served within a certain time (ms)<br>  50%      4<br>  66%      5<br>  75%      5<br>  80%      6<br>  90%      6<br>  95%      6<br>  98%      6<br>  99%      6<br> 100%      6 (longest request)</pre><p>So for the same parameters we took 2 seconds with an RPS of 24.37.</p><h3>How to use this</h3><p>Ideally how you would use this is to improve parts of your code and analyse if your RPS has improved or not. If they have it is a good optimization.</p><p>One point here is that this command does not check for correctness , so if you are returning wrong results but very fast you are essentially getting wrong results.</p><h3>Conclusion</h3><p>Apache Bench is a superb tool if you are sure about your correctness to test your API locally with concurrent requests.</p><p>We are using it to optimize parts of the codebase and evaluating if it is indeed useful for us.</p><p>If you liked this article please consider liking and subscribing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62a6c92fabfb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Issue of exposing service in AWS with K8s Load balancer]]></title>
            <link>https://diptochakrabarty.medium.com/the-issue-of-exposing-service-in-aws-with-k8s-load-balancer-020b02a83859?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/020b02a83859</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Thu, 20 Mar 2025 02:27:01 GMT</pubDate>
            <atom:updated>2025-03-20T02:27:01.016Z</atom:updated>
            <content:encoded><![CDATA[<h4>CLOUD COMPUTING</h4><h4>The annotations required to deploy a load balancer as a service in k8s deployed in AWS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QEZlLSDSRDT43RHC" /><figcaption>Photo by <a href="https://unsplash.com/@mike_van_den_bos?utm_source=medium&amp;utm_medium=referral">Mike van den Bos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So we were deploying our super fast web app in the cloud and wanted to stress test it. There was only one big problem.</p><p>Our app was deployed on a kops cluster in AWS and we needed a mechanism to expose it out to the world in a proper manner.</p><p>And one of the best ways to do that was to expose the service as a Load balancer , however what we saw was that it was not so simple as it sounds.</p><h3>Manifests</h3><p>Firstly lets go over the manifest that we used for our service</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: reddit-service<br>  labels:<br>    app: reddit-app<br>    environment: production<br>spec:<br>  type: LoadBalancer<br>  ports:<br>    - port: 80<br>      targetPort: 8080<br>      protocol: TCP<br>  selector:<br>    app: reddit-app<br>    environment: production</pre><p>Once you deploy this in kops we notice this</p><pre>kubectl apply -f service.yaml</pre><p>However the first thing we noticed was that our service was pending , our load balancer service had received an internal IP address however the external Ip address had not been set.</p><p>Moreover when we checked from our AWS page for new load balancers there were none.</p><p>Here are the steps to take to fix this issue.</p><h3>The annotations</h3><p>Modify the service file to add a few annotations which provides with some extra context on what type of Load balancer you are requesting</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: reddit-service<br>  labels:<br>    app: reddit-app<br>    environment: production<br>  annotations:<br>    service.beta.kubernetes.io/aws-load-balancer-scheme: &quot;internet-facing&quot;<br>    service.beta.kubernetes.io/aws-load-balancer-type: &quot;external&quot;<br>    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: &quot;ip&quot;<br>spec:<br>  type: LoadBalancer<br>  ports:<br>    - port: 80<br>      targetPort: 8080<br>      protocol: TCP<br>  selector:<br>    app: reddit-app<br>    environment: production<br></pre><p>Once you apply this file you will notice a new load balancer available to route requests to your pods using the service.</p><p>Lets now understand what each of them mean.</p><h4>aws-load-balancer-scheme</h4><p>This field specifies AWS if a load balancer is internet facing or for internal services.</p><p>Internet facing makes the load balancer accessible from the internet and was useful for our use case for public facing applications.</p><p>The internal on the other hand is if you need a load balancer for the services within your VPC.</p><h4>aws-load-balancer-type</h4><p>While the above one specified the scheme of the load balancer this specifies the type of load balancer.</p><p>AWS two different types of load balancer NLB and ALB which operate on different network layers.</p><p>If you want a load balancer which processes requests very fast and in a simplistic way NLB is the way to go (next blog incoming on this).</p><p>The values that can be set here are</p><ul><li>external: creates an external load balancer</li><li>nlb-ip: This is deprecated but creates a NLB load balancer with IP based targets instead of instance based targets.</li></ul><h4>aws-load-balancer-nlb-target-type</h4><p>The choices under this annotation are</p><ul><li>Instance: this forwards traffic to the node port of the ec2 instances . The Kube proxy forwards the traffic then to the individual pods.</li><li>Ip: this directs traffic directly to the individual pods. So load balancer forwards traffic directly to the pod which makes it avoid one extra hop.</li></ul><p>Once we specify these the loadbalancer comes up. Not all are needed but the article also guides on how to optimize the load balancer for performance and also fixing the error of load balancer stuck in pending in AWS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=020b02a83859" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The battle of Hogwarts between Elasticity and Scalability]]></title>
            <link>https://diptochakrabarty.medium.com/the-battle-of-hogwarts-between-elasticity-and-scalability-0c57b1176f6a?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c57b1176f6a</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 17:20:05 GMT</pubDate>
            <atom:updated>2025-03-14T17:20:05.144Z</atom:updated>
            <content:encoded><![CDATA[<h4>CLOUD COMPUTING</h4><h4>The difference between elasticity and Scalability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*izyCqhrTZgIsHyWD" /><figcaption>Photo by <a href="https://unsplash.com/@dianamia?utm_source=medium&amp;utm_medium=referral">C Dustin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Cloud computing has affected our daily lives and if you are a developer or student you eventually have to deal with the cloud.</p><p>With the responsibilities of the cloud you need to realise the difference between elasticity and scalability.</p><p>Lets break these down with examples.</p><h3>Elasticity</h3><p>You are running a web app and you are using 10 ec2 instances. Its new year next day and your load increases because the new Pokemon cushion has become very famous and now everyone wants it.</p><p>So your cloud provider automatically adds instances to your system and now customers are able to buy the new cushion.</p><p>Within a week the hype has died down and now as your load decreases so your cloud provider decreases the number of instances.</p><p>This is elasticity where resources are adjusted dynamically as per the demand.</p><h4>Key characteristics</h4><ul><li>Automatic resources are adjusted</li><li>Rapid response to real time workload requirements</li><li>Resources are quickly added or removed</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*41R7QvCjXdTuYvTejXYJIg.png" /><figcaption>How elasticity works</figcaption></figure><h3>Scalability</h3><p>You notice in your webapp start-up which started with 100 concurrent users is increasing by 1000 concurrent users and then to 10k concurrent users.</p><p>Your database is unable to handle this load so you slowly improve the DB by adding more capacity , RAM etc.</p><p>This is scalability , unlike elasticity this is not dynamic but more of a planned thing that you perform.</p><h4>Key characteristics</h4><ul><li>Resources are added in a planned manner</li><li>These are to ensure consistent performance with growing workloads</li><li>A huge focus of this is on performance of the application.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*NEGrd4Ro1nRzoM9b7XA6Kw.png" /><figcaption>How scalability works</figcaption></figure><h3>Real case Human Scenario</h3><ul><li><strong>Elasticity</strong>: A restaurant becomes very busy on a Sunday afternoon so the waiter and owners add extra tables and chairs to accommodate more customers. Later they remove the extra tables.</li><li><strong>Scalability</strong>: The next Saturday is a holiday and the owner anticipates lots of customers so he adds extra tables and chairs for that.</li></ul><h3>Conclusion</h3><p>In hindsight elasticity and scalability look the same however the key difference is that elasticity is more flexible however scalability is more preferable idea for long term as you grow.</p><p>As your load increases as you scale your application you wont need to have sudden bursts of elasticity as you can take up more load.</p><p>Scalability provides better control and customization and sets you up for expansion however elasticity is for that one big day when you become popular suddenly and now everyone wants to use your product suddenly.</p><p>Both are important and useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c57b1176f6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Launch Templates can save you tons of time]]></title>
            <link>https://diptochakrabarty.medium.com/aws-launch-templates-can-save-you-tons-of-time-cbf5cba81b12?source=rss-6a7d00f09c48------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf5cba81b12</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dipto Chakrabarty]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 16:19:16 GMT</pubDate>
            <atom:updated>2025-03-13T16:19:16.096Z</atom:updated>
            <content:encoded><![CDATA[<h4>Cloud Computing</h4><h4>All about AWS launch templates and a basic use</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kHKWIx4R030KZpfd" /><figcaption>Photo by <a href="https://unsplash.com/@karishea?utm_source=medium&amp;utm_medium=referral">Kari Shea</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you have worked with AWS you know how you can bake and create your own AMI to reuse in different instances.</p><p>Well how about you can do something like that for your complete instance.</p><p>That&#39;s where launch templates come in.</p><h3>What are Launch Templates</h3><p>AWS Launch Templates are a capability introduced in 2017 that streamline and simplify the instance launch process for Amazon EC2.</p><p>They store instance configuration parameters so you don’t have to specify them every time you launch an instance.</p><p>This makes it very easy to launch multiple instances of the same kind.</p><p>One good usecase of this is for autoscaling groups where you want AWS to dynamically add your instances.</p><p>This ensures that the new instances</p><ul><li>use the same AMI specified</li><li>Use the same security groups</li><li>are of the same instance type</li></ul><h3>How to create a launch template</h3><p>Under ec2 head over to launch templates and create a launch template</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Etl3TU0YtEnWgLQD7BBaA.png" /><figcaption>Select create</figcaption></figure><p>Set a name and description if you need.</p><p>Now more importantly choose the AMI to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sq4QUQIzx2nXS2GWAxMA-Q.png" /><figcaption>Set name and AMI id</figcaption></figure><p>Next specify the type of instance and the security group to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JO0yBxjU9iPZA_fWK3Ehxg.png" /><figcaption>Set instance type</figcaption></figure><p>You can also allocate how much volume each instance will use.</p><p>Once created you can view your templates.</p><h3>Launching an Instance</h3><p>When you auto scale you wont launch instances like this but for now lets launch an instance manually.</p><p>Click on the launch template and choose action launch instance from the template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3wNY47RToiOvdtniXaPWTA.png" /><figcaption>Create a new instance</figcaption></figure><p>And thats it launch instance and the instance will be up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jc6WNMtxpnUHodxen_nPAg.png" /><figcaption>Launch an instance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3dpPRV6Vu63Pqx4Nq3vJIw.png" /><figcaption>Instance is running</figcaption></figure><h3>Conclusion</h3><p>This is how to create a launch template however its use is much more advanced. In the upcoming blog I will tell how to use it for autoscaling.</p><p>If you liked the content make sure to subscribe and like the post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf5cba81b12" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>