<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marcus Santos on Medium]]></title>
        <description><![CDATA[Stories by Marcus Santos on Medium]]></description>
        <link>https://medium.com/@mvinicius.santos?source=rss-abee91b05c0f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*y0qiKAHePb7kA4Jq.</url>
            <title>Stories by Marcus Santos on Medium</title>
            <link>https://medium.com/@mvinicius.santos?source=rss-abee91b05c0f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:37:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mvinicius.santos/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Real Time E-mail Tracking Solution with AWS Services]]></title>
            <link>https://medium.com/@mvinicius.santos/a-real-time-e-mail-tracking-solution-with-aws-services-246d1bc7e4ee?source=rss-abee91b05c0f------2</link>
            <guid isPermaLink="false">https://medium.com/p/246d1bc7e4ee</guid>
            <category><![CDATA[lucidchart]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[amazon-redshift]]></category>
            <dc:creator><![CDATA[Marcus Santos]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 20:19:20 GMT</pubDate>
            <atom:updated>2018-03-15T20:19:20.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*xTIteYxi-JY-SWy0." /></figure><h3>This article presents the development of an e-mail tracking solution using AWS services to solve a real-life problem faced by a web-based application hosted in AWS.</h3><h3>The challenge</h3><p>The subscription process to a particular online service starts through invites sent via e-mail to its intended users. Several other customer interactions, such as password resets and internal process notifications are handled through email as well. On average, over 1000 e-mails are sent daily. This volume spikes significantly whenever a new major customer starts using the service.</p><p>Despite the application being hosted on <em>AWS (Amazon Web Services)</em>, the email infrastructure initially deployed was pretty traditional, a Linux server on <em>EC2 </em>(Elastic Compute Cloud) with P<em>ostfix</em> handling the messages. All components across the platform were setup to use this server for sending emails. While this approach worked great in the beginning, as the number of clients begun to increase, what seemed like small and manageable problems, quickly got out of hand. Among these were:</p><p>Server resource utilization, more specifically <em>CPU</em> and <em>RAM</em> of the e-mail server had to be monitored and regularly have its allocation upgraded as the volume of messages increased. Storage allocation for <em>EBS</em> (Elastic Block Storage) volumes housing the operation logs also had to be closely monitored and periodically increased in order to cope with the growing message traffic.</p><p>Another aspect which could use improvement was tracking of the messages there were actually sent out. Frequently there would be customers complaining about not receiving emails sent out by the application. The process of searching the application logs for the missing messages started to demand an increasing amount of time and effort from the infrastructure team.</p><p>The application was being impacted by what, in the end, was nothing more than a simple process of sending emails and since its entire infrastructure is deployed to <em>AWS, </em>there should be a better and more efficient way to do that, using a real “cloud approach”.</p><p>After a few days of study and some testing, the following solution was designed to address the problem.</p><p><strong>The Solution</strong></p><p>First, the set of criteria believed to be essential for the scenario was defined:</p><p>1- It must be as reliable as possible. The e-mails should be correctly sent and not bounced by blacklists.</p><p>2- Scalability was an inherent requirement do to the nature of the service. The email sending process could not be impacted by hardware limitations due to increase in the number of messages sent.</p><p>3- Cost effectiveness called for a <em>serverless</em> scenario. The solution had to be implemented using few or preferably no servers at all.</p><p>4- Searchable: Process tracking and message lookup in the log records should be fast and straightforward. All information for the recent past should be available in real time.</p><p>5- It needs to have a low cost. As is always the case, these goals should be accomplished with as little added expenditure as possible.</p><p>Browsing the <em>AWS</em> services portfolio, the first one that stood out was <em>SES</em> (Simple Email Service), a managed email service that is reliable, scalable and simple to use. The first three goals could be met using only one service.</p><p>The appropriated domains were configured in <em>SES</em> and the application development environment was set up to use it. After some testing, all messages sent were successfully delivered.</p><p>There was still a big challenge, though. How to process the email log records and store the information so it could be searched by the customer support staff?</p><p>To address this one, <em>SNS</em> (Simple Notification Service) and <em>AWS Lambda</em> proved to be right services.</p><p>According to AWS documentation:</p><blockquote>Amazon Simple Notification Service (SNS) is a flexible, fully managed <a href="https://aws.amazon.com/pub-sub-messaging/">pub/sub messaging</a> and mobile notifications service for coordinating the delivery of messages to subscribing endpoints and clients.</blockquote><blockquote>AWS Lambda lets you run code without provisioning or managing servers.</blockquote><p>In <em>SNS</em> we configured 3 topics, one for message deliveries, other for bounces and the last one for handling message complaints.</p><p>Three functions were created in <em>AWS Lambda</em> using <em>Javascript</em>, to handle those notification types and configured to trigger when a new<em> SNS </em>notification occurred. Whenever one of these <em>SNS</em> topics triggered a <em>Lambda</em> function, the body of the notification was extracted and saved as an object into an <em>S3</em> (Simple Storage Service) bucket.</p><p>At that point, all email records were processed and stored, but there was still a need for an efficient way to perform searches in this massive repository.</p><p>The solution was to load all the information stored in <em>S3</em> into an <em>Amazon Redshift</em> instance. Since both services integrate with each other, this was a very straightforward process. A <em>Redshift</em> table was created with the relevant fields for the searches and the load process was orchestrated by a small EC2 instance. The load operation occurs every 15 minutes..</p><p>With the information inside a relational database, a <em>BI (Business Intelligence)</em> tool, the option was for one available in AWS Marketplace, could be connected to it and dashboards and reports to properly monitor the the operation were created. There was always the option of direct SQL queries in Redshift as well. The complete diagram of the solution is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NhDi8wwsmK1jYa1i." /></figure><p><strong>Refining the solution</strong></p><p>The solution was working great and meeting expectations but it still wasn’t 100% serverless. There was still a lonely <em>EC2</em> instance orchestrating the transfer from <em>S3</em> to <em>Redshift</em>. In addition, despite being regularly updated, the data still wasn’t available in real time for the support staff.</p><p>In the end of 2015, AWS released a service called <em>Kinesis Firehose</em>. This is a variation of the previously released <em>Kinesis Streams</em>, that captures and loads streaming data into other <em>AWS</em> services, like <em>Redshift</em>. Firehose, unlike <em>Kinesis Streams</em>, will automatically scale on demand.</p><p>This service made it possible to finally eliminate the last <em>EC2</em> instance and move to a totally <em>serverless</em> architecture.</p><p>A single <em>Lambda</em> function was developed to perform the message information processing, instead of the previous three that were being used, and its logic was changed in a way that, instead of putting the log messages in an <em>S3</em> bucket, it would insert the message into a <em>Firehose</em> stream where, once ingested, the record is automatically loaded into <em>Redshift</em>, in real time. No more servers or scheduled processes needed.</p><p>Now, the support staff have instant and real time access to mail tracking data and can provide tracking information to the customers immediately. Here is the diagram for the new solution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jw02pTXH9RixQGOG." /></figure><p><strong>Conclusion</strong></p><p>Nowadays, we should always take scale into consideration when developing applications and the cloud provides a myriad of resources to make that possible.</p><p>This case study shows what an increasing number of users can do to an application, even when just one of its modules is not properly designed to scale.</p><p>When designing for the cloud, always give preference to the use of <em>serverless</em> architectures and managed services when possible. All you have to do is know how they can connect to each other and work together. This will allow you to stop worrying about server management and move your focus to what really matters and that is developing and optimize your code.</p><p>This solution has been operating smoothly with no downtime since it was deployed, it is completely capable of scaling in or out automatically, according to demand and has already processed over 9.5 million messages.</p><p><em>I’d like to thank </em><a href="https://www.linkedin.com/in/felipeamaro/"><em>Felipe Amaro</em></a><em>, who worked with me in the development of this solution and also </em><a href="https://www.linkedin.com/in/mauriciolongo/"><em>Mauricio Longo</em></a><em> for helping me with the revision of this article.</em></p><p><em>All diagrams in this article were created with </em><a href="http://www.lucidchart.com/"><em>Lucidchart</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=246d1bc7e4ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>