<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Asangika Kuruppu on Medium]]></title>
        <description><![CDATA[Stories by Asangika Kuruppu on Medium]]></description>
        <link>https://medium.com/@asangikak?source=rss-315b5683fed8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YuJpMCKCCIjuhLgXek1BlQ.png</url>
            <title>Stories by Asangika Kuruppu on Medium</title>
            <link>https://medium.com/@asangikak?source=rss-315b5683fed8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 14:15:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@asangikak/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS Glue — Troubleshooting Issues]]></title>
            <link>https://aws.plainenglish.io/aws-glue-troubleshooting-issues-e47211ce73ce?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e47211ce73ce</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Mon, 23 Jan 2023 09:01:21 GMT</pubDate>
            <atom:updated>2023-01-23T09:01:21.728Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Glue — Troubleshooting Issues</h3><h4>How to Pinpoint Issues</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2FPlKH_1vYIjerW" /><figcaption>Photo by <a href="https://unsplash.com/@mariana42?utm_source=medium&amp;utm_medium=referral">Mariana B.</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>AWS Glue is used to transform and prepare data for analytics. It can discover structured and semi-structured data. The data source for Glue can be Amazon S3, Amazon Redshift, or various databases running on AWS. You can write Scala or Python code into Glue for your custom ETL jobs.</p><p>If you’ve used Glue jobs in your data pipeline, you might have come across issues from time to time. How do you fix a Glue job issue? In this article, I’ll be guiding on how to narrow down performance issues, out-of-memory issues, or data issues in Glue.</p><p>Here are some steps to narrow down a glue job issue.</p><h4><strong>1. Check the CloudWatch Logs</strong></h4><blockquote>Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real time.</blockquote><p>Enable logs and go to the logs to see if you can identify any errors. <br>If the error message is clear and you need to upgrade your resources then do it. For eg: Out of memory issue</p><h4><strong>2. Identify a step that fails or not executing</strong></h4><p>If you used custom Python or Scala code, add print() or printSchema() to find out the steps that are executing without any issues. This way you can identify the step that fails.</p><p>Let’s say you’ve identified that the Glue job fails after a step. It can happen due to a problem in the data you input or due to a problem in your code (not optimized).</p><h4><strong>3. Check whether the incoming data is clean</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKoxiPJO7XU5KHERvd9-Og.png" /><figcaption>image by author</figcaption></figure><p>You can do the following steps to find out if the data is corrupted.<br>- Read individual files instead of reading all the files.<br>- Break down large files into smaller files.<br>Let’s say you are trying to read multiple structured or semi-structured files from an s3 location. If there are multiple files in the s3 location you read, does this problem apply to all the files? Isolate one file and read it. Are you able to read at least one file without any issues?<br>Let’s say you can read some files without any issues. Take a file that causes the issue and break it into smaller files and save. Then read again. This way you can identify whether the issue is related to file size or not.<br>Once you’ve done the above steps and if the issue prevails for smaller files as well, then it is most likely a data issue.</p><h4><strong>4. Optimize code</strong></h4><p>The glue job has a performance issue due to custom code. It is best to optimize the code. Sometimes optimizing code can fix memory issues as well.</p><p>That’s all! Thanks for reading.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em>, </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em>, and</em></strong><em> </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Build awareness and adoption for your tech startup with </em></strong><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e47211ce73ce" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/aws-glue-troubleshooting-issues-e47211ce73ce">AWS Glue — Troubleshooting Issues</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Through A Scientific Lens]]></title>
            <link>https://medium.com/tea-with-mother-nature/through-a-scientific-lens-40dabfc1adbc?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/40dabfc1adbc</guid>
            <category><![CDATA[nature]]></category>
            <category><![CDATA[sustainability]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[creativity]]></category>
            <category><![CDATA[tea-with-mother-nature]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Tue, 22 Mar 2022 20:47:58 GMT</pubDate>
            <atom:updated>2022-03-22T20:47:58.067Z</atom:updated>
            <content:encoded><![CDATA[<h4>Birds, Bees &amp; Wolves</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KLeZ-zhyvbucBdi-" /><figcaption>Photo by <a href="https://unsplash.com/@pascalvendel?utm_source=medium&amp;utm_medium=referral">Pascal van de Vendel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I put the pen down and leaned back. I stretched the fingers of my hand and made a fist repeatedly until the numbness and tingling in my fingers went away. Tired, I looked away from the large screen in front of me. I’ve been reading notes for the last two hours, trying to solve problems applying various statistical equations.<br>Can we solve any problem using science and mathematics?<br>There are real-world problems that cannot be solved efficiently using statistical techniques. Scientist Warren Weaver first wrote about this in his article “Science and Complexity.”</p><blockquote>“… With a given total of national resources that can be brought to bear, what tactics and strategy will most promptly win a war, or better: what sacrifices of present selfish interest will most effectively contribute to a stable, decent and peaceful world? This problems-and a wide range of similar problems in the biological, medical, psychological, economic, and political sciences-are just too complicated to yield to the old nineteenth~century techniques which were so dramatically successful on two-, three-, or four-variable problems of simplicity. These new problems, moreover, cannot be handled with the statistical techniques so effective in describing average behavior in problems of disorganized complexity.”</blockquote><p>So is there a science to solve these new problems that can be categorized into social, economic, political, and biological systems?</p><p>There is an area of study to systematically discover new knowledge about social conduct, structure, and pattern with computational tools. It tries to analyze then simulate our complex and interconnected natural world through coding.</p><p>Craig Reynolds first developed a simulation in 1986 that can simulate flocks of birds. It is called the Boids model. In other words, the program can imitate a flocking system such as; a flock of birds, a swarm of bees, or a school of fish. You might think it is trivial. When you see birds flying in a triangular shape in the sky, you might think their behavior comes from following the first most bird. There is a top-down hierarchy. But a flocking system is a complex adaptive system in nature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_zoJbODAP9ZgGTyj" /><figcaption>Photo by <a href="https://unsplash.com/@jcraice?utm_source=medium&amp;utm_medium=referral">Julia Craice</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The pattern is created not by an individual but by the collective behavior of individual birds(boid). This phenomenon happens when each bird follows three simple instructions when interacting with its neighbors. That are separation, alignment, and cohesion.<br>Separation: Each bird needs to stay close to others but can’t get too close.<br>Alignment: Each bird needs to align with its neighbors.<br>Cohesion: Each bird is attracted to the center of the crowd.<br>No leader bird conducts the other birds.</p><p>Ant colonies, beehives all show this type of intelligent behavior.</p><p>As Dr. Thomas D. Seeley says in his book “Honeybee Democracy”</p><blockquote>… Choosing the right dwelling place is a life-or-death matter for a honeybee colony…Given the vital importance of choosing a suitably roomy and snug homesite, it is not surprising that a colony’s choice of its living quarters is made not by a few bees acting alone but by several hundred bees acting collectively.</blockquote><p>Honeybees form good decision-making groups. Their effective choices are made in a democratic endeavor and not by the queen bee. A group, when organized well, can bring more options to the problem than an individual. There is a higher probability that someone will come up with an exact solution to their problem. When searching for a new location, a group of bees explores suitable “houses”. Upon arrival, they interact with a few other bees. They share the locations with those other bees. Then others go and check those locations(potential houses) for themselves. This happens repeatedly. Finally, the best “house” among them will get the majority of votes. “BeeSmart Hive Finding” simulates this phenomenon. Simple local rules of interaction resulted in a kind of emergent behavior.</p><p>Another interesting phenomenon was observed in Yellowstone National Park in The US in 1995. The wolves changed the rivers. Yes, they did.</p><p>Now, if you, like me, heard it for the first time, what was the first imagery that came to your mind? Did you imagine a wolf pack running along a riverbank? <a href="https://www.youtube.com/watch?v=ysa5OBhXz-Q">Here</a> is a video of how this happened. In summary, the wolves were absent for a long time. This absence of wolves left a hole in the ecosystem. Then fourteen wolves were reintroduced to the park in 1995. This decision had dramatic consequences as this reestablished the balance of the ecosystem. The ecological effects of the wolves reached the other animals and the environment itself. Erosion decreased, the river banks became more stabilized. It led to the rivers and streams changing course.<br>No simulation has been created for this yet. But, this predator-prey population dynamic is also an example of complex adaptive systems.</p><p>Human systems are complex adaptive systems as well. At the same time, humans are a subset of larger complex adaptive systems in nature. Are you aware of the consequences of your actions in the larger ecosystems? What kind of new behaviors will it create in society? What kind of complexity will it produce in the environment?</p><p>To understand a complex adaptive system, studying individual parts is not enough. You need to observe how the parts interact as an ensemble. Then build simulations. Simulations are a great way to get insights into real-world problems and transfer knowledge.</p><p>I have gained only fundamental knowledge in complex adaptive systems. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40dabfc1adbc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tea-with-mother-nature/through-a-scientific-lens-40dabfc1adbc">Through A Scientific Lens</a> was originally published in <a href="https://medium.com/tea-with-mother-nature">Tea with Mother Nature</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Early Ride]]></title>
            <link>https://medium.com/@asangikak/an-early-ride-b2dbde088d75?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2dbde088d75</guid>
            <category><![CDATA[creativity]]></category>
            <category><![CDATA[short-story]]></category>
            <category><![CDATA[nature]]></category>
            <category><![CDATA[sunrise]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Mon, 08 Nov 2021 13:41:02 GMT</pubDate>
            <atom:updated>2021-11-14T00:20:53.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WO86FQoU0hEqgjEu" /><figcaption>Photo by <a href="https://unsplash.com/@landscapeplaces?utm_source=medium&amp;utm_medium=referral">Federico Bottos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Usually I have breakfast around 11.30. These days I start work around 10 o’clock which means of course I wake up late. A lot later than those early birds.</p><p>But one fateful day, I woke up around 3 o’clock and got ready to go out. I drove far, early in the morning. I had to travel from the highway to reach my destination. I passed the usual woods, rubber estates, rivers, paddy fields and hills that stood on the sides of the road. I saw one of the most beautiful scenery in my life that day. When you travel that early, you can see the sunrise! You might even see Sri Pada just like me. It was so wonderful to see the misty mountains and the idle valleys paint with color through the long ride.</p><p>It starts with a deep purple color spreading over the clouds. Then the sky lights with a deep blue slowly. The brightness and hue changes little by little and the sky changes to a bright blue color. Then the first rays appear in the clear sky. It is golden and you can see stripes of golden yellow. It is picturesque than anything you’ve ever seen. They grow brighter and changes to a luminous orange color as the sun rises. Then the clouds become whiter and the sun is no longer gold. It is a bright yellow ball and the woods that used to be outlines starts to appear removing the black curtains. Their tops scarfed with yellow…Everything becomes normal slowly and you run out of words to describe the usual. Except everybody&#39;s mantra Eat, Work, Sleep and Covid until day is done.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2dbde088d75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Send an Email Using Amazon SES]]></title>
            <link>https://aws.plainenglish.io/send-an-email-using-amazon-ses-b8b5848a93f4?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8b5848a93f4</guid>
            <category><![CDATA[aws-simple-email-service]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[email]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Sat, 06 Nov 2021 17:25:42 GMT</pubDate>
            <atom:updated>2025-02-03T12:46:49.585Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide for sending emails with Amazon SES using a domain.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cza3Amk_fAaWoILd" /><figcaption>Photo by <a href="https://unsplash.com/@ecees?utm_source=medium&amp;utm_medium=referral">erica steeves</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>SES is a cost-effective and scalable email service that can be used to send several email types. There is so much to learn in SES. This is only an introduction. There are two paths to choose when implementing this. Using domain or without using a domain. For the latter, you can refer to <a href="https://medium.com/@deshan.m/aws-email-service-ses-without-a-domain-8f76140c1396">this</a> nice article. Now let’s try to implement using a domain.</p><p>First, let’s create a domain name. We can use <a href="https://aws.amazon.com/route53/pricing/">Route 53</a> to buy a domain name. Route 53 is not free. (But if you delete it within 12 hours of creation you will not be charged!)</p><p>You need to register a domain. Go to Route 53 →Registered domains → Click on Register domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dWLxlUSjDIe5SFm3-U1_Qg.png" /><figcaption>Go to Registered domains</figcaption></figure><p>There is a cost involved in registering a domain. Refer to the <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-register.html">AWS documentation</a> for the details on prices. Give a name you prefer and proceed. When you register a domain, AWS automatically creates a hosted zone that has the same name as the domain. When you register a domain it will take some time to register (mine took about fifteen minutes) meanwhile it will appear in the pending requests. Patience is key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3Jnue1KeD52DglCnp4eiw.png" /><figcaption>Register a domain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4yceqYc6eqlqBJSDGS8cXQ.png" /><figcaption>Domain is registered</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--f3NolTgLOBbIjrKwJ9xQ.png" /><figcaption>Hosted zone</figcaption></figure><p>Now if you go and check your email (the one you gave when registering the domain) you will notice 2 emails. One from Amazon and one from Route 53. Click the link in amazon one and verify your email address.</p><p>Go to Amazon SES → Click on Create identity. Add your domain name and create an identity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h84XztF6uOyg4sd3Z9rRJQ.png" /><figcaption>Create Identity</figcaption></figure><p>Scroll down and click on Use the classic console. (While I was doing this, I noticed a bug. Even if you cancel the popup, the page refreshes and loads the classic console. Does that happen to you? Let me know in the comments). You can also click verify a New Domain → Give the Domain → Use Route 53 → Create Record Sets. Wait till it is verified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fxhDWmnhl9L6kkUZQkw2fw.png" /></figure><p>Now we can use this domain to send emails. For example for “From” email address you can use anything like admin@asangika.link, notification@asangika.link, etc.</p><p>Next, go to Email addresses. Add the email address you want to send the email to. This will be the “To” email address. Since we are in the <strong>sandbox </strong>we need to verify this. Click on verify a new email address. Add your email address. Then AWS will send a verification request to that email. The email subject will be “Amazon Web Services — Email Address Verification Request in region…”. It needs to be verified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yfjIMj0qeHZwI7PDG-cMMA.png" /><figcaption>Add email</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5WjRWY7SGDdXad6QLF_fw.png" /><figcaption>Verify a New Email Address button &amp; Email verified</figcaption></figure><p>Next, let’s create a Lambda function to test this. Paste the below code.</p><pre>import boto3</pre><pre>def lambda_handler(event, context):<br>    # TODO implement <br>    <br>    email = [&quot;ToEmailAddress&quot;] # you can add multiple emails<br>    <br>    ses = boto3.client(&quot;ses&quot;)<br>    try:<br>        <br>        response = ses.send_email(<br>            Source = &quot;admin@asangika.link&quot;,<br>            Destination={<br>                &#39;ToAddresses&#39;: email<br>            },<br>            Message={<br>                &#39;Subject&#39;: {<br>                    &#39;Data&#39;: &quot;AWS SES Email Demo&quot; <br>                },<br>                &#39;Body&#39;: {<br>                    &#39;Text&#39;: {<br>                        &#39;Data&#39;: &quot;This is a test email. \n&quot;<br>                        <br>                    }<br>                }<br>            }<br>        )<br>        print(&quot;Success&quot;)<br>        <br>    except Exception as e:<br>        print(e)</pre><p>Make sure the Lambda Function IAM Role has the following permissions.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ses:SendEmail&quot;,<br>                &quot;ses:SendRawEmail&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:ses:Region:AccountID:identity/*&quot;,<br>                &quot;arn:aws:ses:Region:AccountID:configuration-set/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFXMcsBGHFRW1JnwB9RZqA.png" /><figcaption>Email Received</figcaption></figure><p>This guide walks you through setting up SES for sending emails. That said, dealing with domain verification, IAM roles, and Lambda just to send emails can be a hassle. If you’d rather skip the AWS complexity and get started immediately, something like <a href="https://notify.cx/?utm_source=medium&amp;utm_medium=blog_post&amp;utm_campaign=q1_2025_embed">Notify</a> could be a great alternative — it works out of the box without the setup headaches.</p><p>That’s all! Hope this helps.</p><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8b5848a93f4" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/send-an-email-using-amazon-ses-b8b5848a93f4">How to Send an Email Using Amazon SES</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Send an Email Notification Using AWS Lambda]]></title>
            <link>https://aws.plainenglish.io/send-an-email-notification-using-aws-lambda-c7a24751e969?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7a24751e969</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-sns]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Thu, 04 Nov 2021 16:29:52 GMT</pubDate>
            <atom:updated>2025-02-11T22:43:46.262Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide to Amazon SNS.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_xdzPYwHFy1vBOqW" /><figcaption>Photo by <a href="https://unsplash.com/@hostreviews?utm_source=medium&amp;utm_medium=referral">Stephen Phillips - Hostreviews.co.uk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Amazon Simple Email Service(SES) and Amazon Simple Notification Service (SNS) are two services that you can use to send emails. <br><a href="https://aws.amazon.com/ses/">SES</a> is a cost-effective and scalable email service that can be used to send several email types, including transactional, marketing, or mass email messages. <br><a href="https://aws.amazon.com/sns/?whats-new-cards.sort-by=item.additionalFields.postDateTime&amp;whats-new-cards.sort-order=desc">SNS </a>enables you to send messages to users via SMS, mobile push, and email (Email is only one method of the many methods). With SNS you can send one message to many subscribers. <br>SES and SNS have variations in their costs. If you consider the delivery email, with SES you can send a standard email with all the formatting whereas with SNS you do not have a lot of room for formatting (That is why it is called a notification). So you should consider all these and implement accordingly.</p><blockquote>👉 If you need more control over formatting and deliverability, SES is the better choice. But SES also comes with a convoluted setup and documentation. If you’d rather skip the AWS headaches, you could check out a tool like <a href="https://notify.cx/?utm_source=medium&amp;utm_medium=blog_post&amp;utm_campaign=q1_2025_embed">Notify</a>. It works out of the box and lets you send 5,000 emails/month for free.</blockquote><p>This is an activity you can try using AWS Lambda and SNS. The task is to send an email notification using AWS Lambda.</p><p>First, you need to go to Amazon SNS and create a topic. You should create a topic, create a subscription and then publish to a topic. Then all the subscribers will get that notification(to Gmail or any other mail address).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8pTUN5spXEdMdxySX-iq7Q.png" /><figcaption>Create Topic screen</figcaption></figure><p>Choose a topic type (standard).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2asA5CAIUpPDtF-ozcMSvA.png" /><figcaption>Choose a topic</figcaption></figure><blockquote><a href="https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sns-introduces-fifo-topics-with-strict-ordering-and-deduplication-of-messages/">FIFO topics</a> are used in combination with Amazon Simple Queue Service (Amazon SQS) FIFO queues, to build applications that require messages to be sent and processed in a strict sequence and without duplicates.</blockquote><p>SNS FIFO is a new feature. Read more <a href="https://medium.com/swlh/amazon-sns-now-has-fifo-e87797f5e3c1">here</a>.</p><p>The display name will be the “from” address. You can change this later if you wish.</p><p>Next, create a subscription. Select “email” as the protocol. Give your preferred email address to the endpoint. You can only give one email address at a time. If you want to send a message to multiple users, you have to create multiple subscriptions. Once the subscription is created, you’ll get an email to subscribe to the topic. Email Subject: AWS Notification — Subscription Confirmation. Click it to confirm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qPrnKuRJQhxiMU9rP5qK0Q.png" /><figcaption>Create Subscription</figcaption></figure><p>Get the SNS Topic ARN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfDJNHjjiuHmhuCzyw7Aww.png" /></figure><p>Create a Lambda function. Paste the below code. Replace the snsArn with your ARN.</p><pre>import boto3</pre><pre>def lambda_handler(event, context):<br>    # TODO implement<br>    <br>    client = boto3.client(&#39;sns&#39;)<br>    snsArn = &#39;arn:aws:sns:Region:AccountID:TestTopic&#39;<br>    message = &quot;This is a test notification.&quot;<br>    <br>    response = client.publish(<br>        TopicArn = snsArn,<br>        Message = message ,<br>        Subject=&#39;Hello&#39;<br>    )</pre><p>One last check. Does your Lambda IAM role have AWS managed policy <a href="https://console.aws.amazon.com/iam/home#/policies/arn%3Aaws%3Aiam%3A%3Aaws%3Apolicy%2FAmazonSNSFullAccess">AmazonSNSFullAccess</a>? Add it and test. That’s all!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zL97v-dXDUS08J7G8E7Uxg.png" /><figcaption>Email Received.</figcaption></figure><p>Let’s assume your lambda function is triggered by some other event. That event output is a <a href="https://www.w3schools.com/js/js_json_intro.asp">JSON </a>string as follows and you want to capture it. Then add it to the email. You have to do a simple change.</p><pre>{&quot;name&quot;:&quot;John&quot;, &quot;age&quot;:30, &quot;car&quot;:&quot;null&quot;}</pre><pre>import boto3</pre><pre>def lambda_handler(event, context):<br>    # TODO implement<br>    <br>    client = boto3.client(&#39;sns&#39;)<br>    snsArn = &#39;arn:aws:sns:Region:AccountID:TestTopic&#39;<br>    message = &quot;This is a test notification.&quot;<br>    <br>   <strong> name = event.get(&quot;name&quot;)</strong><br>    <br>    response = client.publish(<br>        TopicArn = snsArn,<br>        Message = message ,<br>        Subject=&#39;Hello &#39; + <strong>name</strong><br>    )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-rnRit79M5Z8oRxt0OKUGg.png" /><figcaption>Test with JSON data</figcaption></figure><p>That’s all! Thanks for reading.</p><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7a24751e969" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/send-an-email-notification-using-aws-lambda-c7a24751e969">How to Send an Email Notification Using AWS Lambda</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Copy a File Using AWS Lambda]]></title>
            <link>https://aws.plainenglish.io/how-to-copy-a-file-using-aws-lambda-a64eb9d0c902?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a64eb9d0c902</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Tue, 02 Nov 2021 14:26:09 GMT</pubDate>
            <atom:updated>2021-11-04T08:05:06.024Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide on copying from source S3 to destination S3.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nB9fot16w4SO4En5" /><figcaption>Photo by <a href="https://unsplash.com/@freestocks?utm_source=medium&amp;utm_medium=referral">freestocks</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>AWS Lambda is a serverless, event-driven compute service. Serverless does not mean there are no servers. It means you don’t manage or provision them. Someone else does that job for you. Event-driven means it uses events to trigger and communicate between services. When using lambda you just write the function and AWS manages the rest for you.</p><p>It is integrated with many programming languages such as Python, Node.js, Java, C# and Golang. So you can pick the language you prefer. It is that flexible.</p><p>Amazon S3 is a cloud object storage.S3 allows you to store objects in what they call “buckets”. A bucket is like a directory. It can be created in a particular region. But S3 bucket names are unique globally. When I said it is <em>like </em>a directory I mean it. Because inside a bucket there are no folders. It has a logical arrangement similar to folders separated by “/”. Bucket objects or files are always referenced by a key. The key is the full path.</p><p>This is a simple activity you can try in AWS. Here you’d be using two AWS Services: Lambda and S3. As the first task let’s copy a file in the same S3 bucket. The code is simple.</p><p>Create the S3 bucket and add an object. I assume that you have an object called “script.py” in the following source path. So the object key here is the entire “mybucket1/source/script.py”.</p><pre>mybucket1/source/script.py</pre><p>You want the destination path to be</p><pre>mybucket1/destination/script.py</pre><p>Create a Lambda function in the same region as the bucket is in.</p><pre>import boto3</pre><pre>def lambda_handler(event, context):<br>   <br>    # Creating the connection with the resource<br>    s3 = boto3.resource(&#39;s3&#39;)<br>    # Declaring the source to be copied<br>    copy_source = {&#39;Bucket&#39;: &#39;mybucket1&#39;, &#39;Key&#39;: &#39;source/script.py&#39;}<br>    bucket = s3.Bucket(&#39;mybucket1&#39;)<br>    # Copying the files to another folder<br>    bucket.copy(copy_source, &#39;destination/scriptCopy.py&#39;)</pre><p>Paste the above code and deploy the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKKemIwSuDP3NiZU183Hzg.png" /><figcaption>Save and Deploy Lambda Function</figcaption></figure><p>When you create a Lambda function it creates an execution role with basic permissions (If you did not change anything). Make sure the Lambda function IAM Role has the following permissions before you test the function. Go to Configuration-&gt;Permissions. Open the Execution Role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MxcU05bAKCQrQY7dC06Lmw.png" /><figcaption>IAM Role Lambda Function</figcaption></figure><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:GetBucketLocation&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::mybucket1/*&quot;,<br>                &quot;arn:aws:s3:::mybucket1&quot;<br>            ]<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor1&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3-object-lambda:GetObject&quot;,<br>                &quot;s3-object-lambda:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3-object-lambda:Region:AccountID:accesspoint/*&quot;<br>        }<br>    ]<br>}</pre><p>As the second task let’s copy a file in one S3 bucket to another S3 bucket. The code is simple.</p><p>You have to update one line.</p><pre>bucket = s3.Bucket(&#39;mybucket2&#39;)</pre><p>Update the correct permission to the other bucket too.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:GetBucketLocation&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::mybucket1/*&quot;,<br>                &quot;arn:aws:s3:::mybucket1&quot;,<br>                &quot;arn:aws:s3:::mybucket2/*&quot;,<br>                &quot;arn:aws:s3:::mybucket2&quot;<br>            ]<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor1&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3-object-lambda:GetObject&quot;,<br>                &quot;s3-object-lambda:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:s3-object-lambda:Region:AccountID:accesspoint/*&quot;<br>        }<br>    ]<br>}</pre><p>That’s all! Hope this helps.</p><p>Thank you for reading.</p><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a64eb9d0c902" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/how-to-copy-a-file-using-aws-lambda-a64eb9d0c902">How to Copy a File Using AWS Lambda</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How  to Enable Spark UI Using Docker for a Beginner]]></title>
            <link>https://aws.plainenglish.io/in-a-previous-story-i-gave-an-introduction-to-aws-glue-monitoring-d083389ca212?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d083389ca212</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Sun, 31 Oct 2021 19:05:45 GMT</pubDate>
            <atom:updated>2021-11-03T12:51:59.657Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Enable Spark UI Using Docker for a Beginner</h3><h4>A beginner’s guide to enabling Spark UI using Docker.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TKgOuIdsjJsm9yaM" /><figcaption>Photo by <a href="https://unsplash.com/@virgin?utm_source=medium&amp;utm_medium=referral">IIONA VIRGIN</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In a previous story, I gave an introduction to AWS Glue monitoring. If you haven’t read it already read from <a href="https://medium.com/@asangikak/enable-monitoring-in-aws-glue-a-beginners-guide-151c7910edcf">here</a>.</p><p>Now as I mentioned, there are two ways to enable Apache Spark web UI. Docker and CloudFormation. Yeah, I did not share many details on <em>how to enable it,</em> I know.</p><p>It was something I struggled to implement. I understood the CloudFormation method but trust me I had no knowledge of Docker at that time and I did not understand a thing in that article! If you are like me, I am here to tell you “Don’t give up”. Because I figured out how to implement it and it is a piece of cake.</p><p>So here is <em>how to enable Spark web UI in AWS using docker with zero docker knowledge. </em>To make this black magic work you need only two reference links and an AWS account. Plus it is cheap!</p><p>First, you need to create an EC2 account. A free tier Amazon Linux/Amazon Linux 2 AMI is enough. When you create it pick a region that is close to you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEIOZxV8SGz3-9CZO8-iyw.png" /><figcaption>Create EC2 instance</figcaption></figure><p>While creating the instance, paste the following script in the user data section. This will install Docker in the EC2. If you want to dig deeper, I got this from <a href="https://medium.com/appgambit/part-1-running-docker-on-aws-ec2-cbcf0ec7c3f8">here</a>.</p><pre>#! /bin/sh<br>yum update -y<br>amazon-linux-extras install docker<br>service docker start<br>usermod -a -G docker ec2-user<br>chkconfig docker on</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2NmH5yjUxiVfV9d6jpRMg.png" /><figcaption>User data and region</figcaption></figure><p>Now that your EC2 instance is ready to run Docker containers, you need to have the images in the right place. Go to EC2 instances and connect to your instance. You can easily select the instance then Actions-&gt; Connect.</p><p>Next, you need to create the Docker file. A Docker file is a text file where you write instructions to build an image. Create a folder called Images using the following command.</p><pre>$ mkdir images</pre><p>Move inside the “images” folder</p><pre>$ cd images</pre><p>Create a file named Dockerfile inside the images folder</p><pre>$ nano Dockerfile</pre><p>Put the commands inside the Dockerfile. This is where you need to read <a href="https://towardsdatascience.com/glue-spark-ui-6dab088929d9?gi=9ecb0c859b35">the second reference</a>. Copy and paste the “<a href="https://gist.github.com/jnshubham/7fa2876b9d8d4f54e40f590c56166daf#file-glue_sparkui_dockerfile">glue_sparkui_dockerfile</a>” code block into your Dockerfile. Save nano command using Ctrl+O and exit using Ctrl+X. Follow the instructions in the above article and complete up to “Accessing the history server from EC2”. You are almost there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CMoL-IPdWw4rrLrNlc80Dg.png" /><figcaption>Dockerfile</figcaption></figure><p>Now try to access the Spark web UI. Go to your EC2 instance and copy the Public IPv4 address. add port: 18080 at the end of it and paste it in a new tab. The history server will show the spark UI for the glue jobs. If you have successful logs Great! you’ve done it. However, If you currently do not have any logs you will see something like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IKs0LXaQ1lfT_W1QyyiU3A.png" /><figcaption>SparkUI</figcaption></figure><p>You’ve enabled the Spark Ui in the glue job. You gave an S3 bucket location to the logs. So why are there no logs? First, make sure the s3 location is correct. Run the glue job once and then go and check the s3 bucket. If there are no logs still, there are two possibilities.</p><ol><li>The execution time of the Glue job is less than 1 min.</li><li>Glue Job IAM Role does not have permission to write logs to the S3 bucket.</li></ol><p>The <a href="https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html">AWS documentation</a> mentions that <em>Every 30 seconds, AWS Glue flushes the Spark event logs to the Amazon S3 path that you specify. </em>Therefore it is better to have the Glue job run for more than 1 min. (Besides what kind of performance tuning are you going to do in a job that runs less than 60s?)</p><p>If the issue is in the Glue job IAM Role, make sure you have the following S3 and logs permissions.</p><pre>{<br> “Version”: “2012–10–17”,<br> “Statement”: [<br> {<br> “Sid”: “VisualEditor0”,<br> “Effect”: “Allow”,<br> “Action”: [<br> “s3:PutObject”,<br> “s3:GetObject”,<br> “s3:ListBucket”<br> ],<br> “Resource”: [<br> “arn:aws:s3:::BucketName/*”,<br> “arn:aws:s3:::BucketName”<br> ]<br> }<br> ]<br>}</pre><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;glue:StartJobRun&quot;,<br>                &quot;glue:GetJobRun&quot;,<br>                &quot;glue:BatchStopJobRun&quot;,<br>                &quot;glue:GetJobRuns&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor1&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;logs:PutLogEvents&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:logs:Region:AccountID:log-group:*:log-stream:*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor2&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:CreateLogGroup&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;arn:aws:logs:Region:AccountID:log-group:*&quot;<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcmN_KSnOJwsWezCiZvWUA.png" /><figcaption>Spark UI with logs</figcaption></figure><p>That’s all! Hope this helps.</p><p><em>More content at </em><a href="http://plainenglish.io/"><em>plainenglish.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d083389ca212" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/in-a-previous-story-i-gave-an-introduction-to-aws-glue-monitoring-d083389ca212">How  to Enable Spark UI Using Docker for a Beginner</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enable Monitoring in AWS Glue — A Beginner’s Guide]]></title>
            <link>https://medium.com/@asangikak/enable-monitoring-in-aws-glue-a-beginners-guide-151c7910edcf?source=rss-315b5683fed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/151c7910edcf</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Asangika Kuruppu]]></dc:creator>
            <pubDate>Sat, 30 Oct 2021 06:52:06 GMT</pubDate>
            <atom:updated>2021-10-30T06:53:40.249Z</atom:updated>
            <content:encoded><![CDATA[<h3>Enable Monitoring in AWS Glue — A Beginner’s Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ETIT_iK_DEcc2YEZ" /><figcaption>Photo by <a href="https://unsplash.com/@annaelizaearl?utm_source=medium&amp;utm_medium=referral">Anna Earl</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>AWS Glue is a system for building table definitions and performing ETL (Extract, Transform and Load) automatically. You can write your own code to transform the data in Glue using Python or Scala.</p><p>Depending on the processing load AWS Glue tasks can run for hours. You can provision additional DPUs( Data Processing Units) to improve the performance of underlying Spark Jobs. You can enable Job metrics when creating the glue job or afterwards. AWS documentation has more details that you can refer through <a href="https://docs.aws.amazon.com/en_us/glue/latest/dg/add-job.html">this</a>. You can enable Job metrics in your Glue job to figure out the maximum capacity of DPUs you need. Refer <a href="https://docs.aws.amazon.com/glue/latest/dg/monitor-debug-capacity.html">this</a> for more details on metrics. However, Glue 2.0 does not support ExecutionAllocationManager metrics which means in the “Job Execution: Active Executors, Completed Stages &amp; Maximum Needed Executors” Graph, no data will be displayed in Number of Active Executors and Number of Maximum Needed Executors.</p><p>So let’s move on to the other options.</p><p><a href="https://medium.com/swlh/my-top-10-tips-for-working-with-aws-glue-a1cd880961fe">This</a> article describes how to enable continuous logging and access the logs. Quite useful but CloudWatch logs do not give much insights on performance.</p><p>The third option is to enable Apache Spark web UI. you can read more in the documentation <a href="https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html">here</a>. There are two ways to implement this. Using AWS CloudFormation and using Docker.</p><p><strong>Using Docker</strong></p><p>I found an <a href="https://medium.com/beardydigital/visualising-aws-glue-history-in-spark-ui-history-server-bf55c0f5f783">article</a> written on this topic. So according to this docker approach is easier to implement.</p><p><strong>Using CloudFormation</strong></p><p>For those of you who are experienced in CloudFormation, follow <a href="https://docs.aws.amazon.com/en_en/glue/latest/dg/monitor-spark-ui-history.html">this</a> article. We need to set up an EC2 instance to host the history server. The benefit of history server is we can view our metrics even after the glue job has run.</p><p>To create the stack you have to give some parameters. The documentation describes them clearly. Note that there is a privacy concern when giving the IP address range<strong>. </strong>If you give 0.0.0.0/0 it will be accessible to every one.</p><p>Once you create the CloudFormation stack and try to access your public url, it will give a warning saying the connection is not secure. If you accept and proceed, it will send a notification to your aws account email.</p><p>What are your experiences in approaching a performance monitoring issue in AWS Glue? Do let me know in the comments. That will help someone else.</p><p>References:</p><p><a href="https://aws.amazon.com/premiumsupport/knowledge-center/glue-job-running-long/">Troubleshoot AWS Glue job running for a long time</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=151c7910edcf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>