<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tim Specht on Medium]]></title>
        <description><![CDATA[Stories by Tim Specht on Medium]]></description>
        <link>https://medium.com/@tspecht?source=rss-808221e7725d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lcqdZSGEDhA47XjDK6Vrwg.jpeg</url>
            <title>Stories by Tim Specht on Medium</title>
            <link>https://medium.com/@tspecht?source=rss-808221e7725d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:39:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tspecht/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evolving alongside your tech stack]]></title>
            <link>https://medium.com/dubsmash-engineering/evolving-alongside-your-tech-stack-daf50abaaf21?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/daf50abaaf21</guid>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Mon, 21 Oct 2019 12:54:13 GMT</pubDate>
            <atom:updated>2019-10-21T12:54:13.342Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.heroku.com/podcasts/codeish/39-evolving-alongside-your-tech-stack">39. Evolving Alongside your Tech Stack - Podcasts | Heroku</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=daf50abaaf21" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/evolving-alongside-your-tech-stack-daf50abaaf21">Evolving alongside your tech stack</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Realm to build a globally scaling iOS application]]></title>
            <link>https://medium.com/dubsmash-engineering/using-realm-to-build-a-globally-scaling-ios-application-bad863273b43?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bad863273b43</guid>
            <category><![CDATA[realm]]></category>
            <category><![CDATA[meetup]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Fri, 22 Dec 2017 16:46:00 GMT</pubDate>
            <atom:updated>2017-12-22T16:46:11.272Z</atom:updated>
            <content:encoded><![CDATA[<p>Last month I gave a talk at the <a href="https://www.meetup.com/ReactNYC/">i</a>OSOHO meetup about scaling Dubsmash&#39; iOS application using Realm. I gave a small introduction into Realm, it&#39;s basic features and shared some more advanced techniques we used to integrate it into our applications.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F245249509&amp;dntp=1&amp;url=https%3A%2F%2Fvimeo.com%2F245249509&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F670028815_1280.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1a50edcafa4240c299490c55f8346df/href">https://medium.com/media/e1a50edcafa4240c299490c55f8346df/href</a></iframe><p>Slides:</p><p><a href="https://docs.google.com/presentation/d/1jeQZjuADzR1DCu7WZbk9V29FvP9qmhNL6K9kVIo03A8/edit?usp=sharing">[PUBLIC] iOSOHO - Realm</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bad863273b43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/using-realm-to-build-a-globally-scaling-ios-application-bad863273b43">Using Realm to build a globally scaling iOS application</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Dubsmash uses Lambda@Edge to serve dynamic SPA using S3]]></title>
            <link>https://medium.com/dubsmash-engineering/how-dubsmash-uses-lambda-edge-to-serve-dynamic-spa-using-s3-1ba865d34512?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ba865d34512</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[cloudfront]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Thu, 05 Oct 2017 18:50:48 GMT</pubDate>
            <atom:updated>2017-10-05T18:52:06.309Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Od3LxD-rv-ECIABZlGMqrA.png" /></figure><p>When it comes to implementing and delivering globally available web applications, the question of where and how to run or host it usually requires one to set up various different moving parts. This leads to overhead in maintenance and most often also to increased costs.</p><p>In order to mitigate these problems AWS S3 added static website hosting functionality quite some time ago which enables users to simply store HTML files within S3 and have them automatically served by AWS upon request from a client without needing any servers or similar infrastructure. Adding CloudFront in front of this serving layer adds fast global distribution and advanced caching to it.</p><p>This solution is especially interesting for hosting Single Page Applications (SPA) as all that is needed is a simple index.html file which includes the necessary JavaScript bundle. Since all S3 is hosting is a single HTML file there is no possibility for advanced Node.js set ups to deliver the page. This became a problem for our team when we wanted to add Facebook sharing functionality to our existing web application which requires the rendered HTML to contain dynamic OpenGraph metadata in order to render engaging previews when sharing inside Facebook. As the application was a SPA and not all possible URLs were known before, due to the dynamic content and routing (e.g. not all possible HTML files could have been generated on build time by the CI / CD system), we needed to find a new solution to render dynamic pages without sacrificing any or too much of the scalability, pricing and ease-of-use of the current static solution.</p><h3>Why Dubsmash chose Lambda@Edge</h3><p>Before we went ahead and implemented Lambda@Edge we of course had to answer a couple of questions on how we wanted to handle things:</p><ul><li><strong>Running a standard Express / Node.js server on Heroku or Google Cloud Platform</strong>: This is quite an expensive solution since you need to provision servers that run 24/7 as well as redundancy to make sure the site would still be reachable if one should fail. This solution would have added quite some overhead in maintaining the site as well as increased the cost. On the plus side, the setup is well known and easy to do.</li><li><strong>Do we want a fully-fledged SSR (Server-Side Rendering) solution set up? </strong>This can introduce <a href="https://hackernoon.com/server-side-rendering-with-create-react-app-1faf5a9d1eff">quite some additional complexity</a> with managing routes, data &amp; user-state and would require us to run a full Node.js server setup.</li><li><strong>How important is caching to us?</strong> We could run a Node.js server inside Lambda to handle all the possible scalability issues for us but would need to cache ourselves or find some other hacky solution.</li></ul><p>After going through all of the above considerations, we opted to choose Lambda@Edge. We believed this route would get the optimal trade-off between having an easy solution that needs no maintenance (e.g. minimal changes to the existing application while not needing to maintain any servers or infrastructure) while still making use of all the advanced caching and distribution mechanisms CloudFront provides. Also we opted to not implement fully-fledged SSR to avoid having to run any Node.js server at all. Since we are already using Lambda to serve the site we can easily expand this in the future and simply use Lambda as a custom origin behind CloudFront.</p><h3>What is Lambda@Edge?</h3><p><a href="http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html">Lambda@Edge</a> is an extension of Lambda and enables you to execute custom code inside the usual Lambda environment right on, or more precisely, inside the CloudFront edge locations. As these invocations are triggered by CloudFront events for both Viewer (e.g. client is requesting a CloudFront resource) and Origin (e.g. Cache-Miss on CloudFront and the Origin is getting requested) events it enables for a customization of CloudFronts behavior while still making sure you get all the benefits of Lambdas scalability and speed.</p><p>Possible use-cases could be:</p><ul><li>Inspecting requests for certain headers to rewrite the response as part of an A/B test (🍪 anyone?)</li><li>Execute authentication right at the edge by either validating headers or calling another service to verify credentials</li><li>Deliver content based on the users viewport / user-agent to customize experiences for certain devices or screen sizes</li><li>Customize response by doing additional network requests. This could be useful to generate customized log-in or onboarding pages for example.</li></ul><h3><strong>Implementation</strong></h3><p>Since we opted to not implement a full SSR solution we needed to come up with an easy but maintainable workaround to get dynamic metadata tags into our websites <em>HEAD</em> element.</p><p>After some initial whiteboarding and experimentation we decided to check on every Origin request if it was for a page that needed the tags or not with a simple RegEx. The expression would not only serve as a check if the page should be altered but also to extract the necessary UUID fields to facilitate the lookup. Once this was done we would fetch the necessary data from an internally used service, generate the <em>&lt;meta&gt;</em> tags and replace them into the document. Since CloudFront doesn&#39;t allow Lambda@Edge to alter the response it received from the Origin we would have to download the file from CF itself again (which will usually be cached already). <em>Why this behavior exists is beyond my understanding, so if anyone working on AWS ever reads this please feel free to reach out!</em></p><p>Once all the different pieces of content are fetched, the final response is getting built and the callback is called with not the Origin response but the new response and thus rendered as the response to the caller by CloudFront.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/831e41f7ad9f36ac4c738abc60f74ec6/href">https://medium.com/media/831e41f7ad9f36ac4c738abc60f74ec6/href</a></iframe><h3>Limitations &amp; Problems</h3><p>While implementing the solution showcased above and iterating on it we ran into some problems we&#39;d like to share with you. A complete list of all the limitations can be found <a href="http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html">here</a>.</p><ol><li><strong>Developing a solution for Lambda@Edge is painfully slow!</strong> While the integration right on the edge locations provides you with usually blazing fast execution times, deploying changes to your code takes ages. Since you are altering the CloudFront distribution (especially when working with the <a href="https://github.com/serverless/serverless">Serverless</a> framework) you need to wait for all the edge locations to be readily deployed which can easily take 15–20 minutes.</li><li><strong>Error messages are not returned to the caller</strong>. In case you have any error that is raised during the execution of your Lambda function there is little way to see it except using CloudWatch Logs. Since the functions are replicated to all the regions the logs are stored in the region the function is executing in which can lead to quite some confusion and make logs hard to find.</li><li><strong>You can&#39;t use custom error pages when you want to check URIs.</strong> As described <a href="http://aserafin.pl/2016/03/23/react-router-on-amazon-s3/">here</a> it&#39;s a common practice to create a custom error page rule that simply renders out your index.html instead of a 404 response. This is needed to make sure in any case your app gets rendered and React-Router can handle the routing within the app to render the actual content. If you want to only inject meta-data for some URIs as we wanted to do it this behavior is not usable anymore. As soon as your Origin Response trigger is executed CloudFront will have rewritten the request.uri property to /index.html already making it impossible for you to check against the URI the user actually requested. That&#39;s why we removed the custom error behavior and instead added a second Lambda function on the Origin Request trigger that would copy the initial URI to a custom header and then rewrite the request to /index.html to keep the remaining behavior.</li></ol><p>If you would like to work with Lambda, React and other cool state-of-the-art technologies on both frontend- and backend-side, <a href="http://dubsmash.com/jobs">please get in touch</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ba865d34512" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/how-dubsmash-uses-lambda-edge-to-serve-dynamic-spa-using-s3-1ba865d34512">How Dubsmash uses Lambda@Edge to serve dynamic SPA using S3</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to GraphQL and React]]></title>
            <link>https://medium.com/dubsmash-engineering/introduction-to-graphql-and-react-afb80aa47bca?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/afb80aa47bca</guid>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Mon, 02 Oct 2017 17:44:01 GMT</pubDate>
            <atom:updated>2017-10-02T17:54:47.763Z</atom:updated>
            <content:encoded><![CDATA[<p>Last week I gave a talk at the <a href="https://www.meetup.com/ReactNYC/">ReactNYC</a> meetup about GraphQL and React. I outlined how GraphQL works, what the benefits are and gave some examples on how to get started using it together with React.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FF7iwWpeqiTE%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DF7iwWpeqiTE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FF7iwWpeqiTE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a368d13e2f15e651a32dd64cd99b1e9e/href">https://medium.com/media/a368d13e2f15e651a32dd64cd99b1e9e/href</a></iframe><p>Slides:</p><p><a href="https://docs.google.com/presentation/d/1VAtW6BLByTwGJVZ_AmSQuAc6tPGmDj6cdd_CI-bEjkw/edit">ReactNYC - GraphQL</a></p><p>If you would like to work with GraphQL, React and other cool state-of-the-art technologies on both frontend- and backend-side, <a href="http://dubsmash.com/jobs">please get in touch</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afb80aa47bca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/introduction-to-graphql-and-react-afb80aa47bca">Introduction to GraphQL and React</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bi-directional pagination using GraphQL & Relay]]></title>
            <link>https://medium.com/dubsmash-engineering/bi-directional-pagination-using-graphql-relay-b523c919c96?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b523c919c96</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[relay]]></category>
            <category><![CDATA[apollo]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[dubsmash]]></category>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Fri, 28 Jul 2017 12:21:16 GMT</pubDate>
            <atom:updated>2017-07-28T12:21:16.870Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*EOMP0V69RZ5xChG5pRoFyA.png" /></figure><p>Many applications today— both web and mobile — are displaying some sort of data of arbitrary length to their users e.g. a news feed, a list of blog posts, search results or similar. More often than not this data is too much for the application to render all at once on client-side or for the database to return all records when the data is fetched, the response would simply be too big and long to render and the time until the data is visible to the user would drastically decrease the users experience. In addition, most developers will want to display some sort of footer on the bottom of their page also making long lists also a simple UX issue.</p><p>In order to display those list-like user interfaces, modern applications usually rely on different kinds of pagination to fetch and display their data. Most modern web-frameworks both on the backend as well as the frontend side have built-in support for these kind of interactions.</p><h3>How does pagination work in GraphQL &amp; Apollo?</h3><p>GraphQL itself doesn&#39;t need any special handling for pagination on a specification level. All the necessary mechanisms like attributes &amp; nodes are already part of the GraphQL standard and can easily be used to implement pagination mechanisms.</p><p>However, there are 3 common approaches that are usually used to implement pagination using GraphQL (and in general):</p><h4>#1: Offset-based pagination</h4><p>Offset-based pagination is the most common kind of pagination. This stems mostly from it being well-known from pre-GraphQL-times and it being very easy to implement when the backend servicing the API requests is using SQL to fetch the data. <a href="https://www.postgresql.org/docs/8.2/static/queries-limit.html">https://www.postgresql.org/docs/8.2/static/queries-limit.html</a></p><p>In order to paginate based on offsets, clients are sending the index of the item in the list on the database-level which they want to start reading from. Also the amount of items (or limit) they want to have returned can be sent or the server picks a default one.</p><p>The backend implementation can use standard SQL LIMIT queries to fulfill the requests while making sure that the page size isn&#39;t too big to avoid unnecessary high load on a database-level.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02c2f6e4a0994272fac873cdd88dc6bc/href">https://medium.com/media/02c2f6e4a0994272fac873cdd88dc6bc/href</a></iframe><p>Note that in the example above both offset and limit are optional variables. If you don&#39;t send them the server will paginate from the beginning and most-likely will apply a default limit.</p><h4>#2: Cursor based</h4><p>Cursor-based pagination is using a so called cursor to keep track of the element in the list where reading should start (instead of a simple index inside the list for offset-based pagination).</p><p>Cursors can be anything the backend understands. The most simple example for a cursor would be the database-ID of an object (which would make the cursor-based pagination very similar to the offset-based one) but can be more complex. In production, cursors are mostly strings that often contain not only the database-ID of the starting object but also an identifier of some sort indicating how the results were sorted initially so the order can be preserved properly.</p><p>Clients themselves don&#39;t care what the cursor is or what it means to the backend, they simply store it after receiving a response and include it in any subsequent requests for following pages to indicate the starting point of the expected response.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/256670bc27c25d93ee35fe3ffb76fb12/href">https://medium.com/media/256670bc27c25d93ee35fe3ffb76fb12/href</a></iframe><h4>#3: Relay-style cursor based</h4><p>Relay-style cursor-based pagination is basically a flavor of the cursor-based pagination described above. Relay makes a couple of opinionated assumptions about pagination which are reflected in this style of pagination.</p><p>The main difference to classical cursor-based pagination is the response format . A pageInfo object is included in the response which contains the connections cursors for the first &amp; last objects in addition to a hasNextPage and hasPreviousPage flags that can be used to conveniently check if there are more pages to load. In addition, the objects that are paginated themselves are wrapped in an edges attribute which contains node attributes wrapping the objects themselves. <a href="https://facebook.github.io/relay/graphql/connections.htm#sec-Pagination-algorithm">The full spec can be found here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cdc6e9cf7b1edfb867341a76bc874e3/href">https://medium.com/media/4cdc6e9cf7b1edfb867341a76bc874e3/href</a></iframe><p>Note that the actual content about the Pokemòns we want to fetch is wrapped in a node object which itself is wrapped in an edges object. The pageInfo is returned as a separate object containing all the necessary information for the pagination itself.</p><h3>Why doesn&#39;t Relay like bi-directional pagination?</h3><p>Relay was implemented by Facebook with infinite scrolling use-cases in mind. A perfect example for this is Facebooks own newsfeed which provides the user with an endless scrolling experience continuously fetching more content while scrolling down.</p><p>While this is a perfect use-case many developers find themselves wanting to implement a more classical pagination experience. This may include “Previous Page” &amp; “Next Page” buttons for the user to navigate or a page-number based navigation as found on Googles search results page. Many server-side GraphQL implementations like graphene-django (NSERT LINKT) or similar frameworks for other languages implement their pagination Relay-style since it&#39;s such a wide-spread and well-known framework.</p><p><a href="https://facebook.github.io/relay/graphql/connections.htm#sec-Pagination-algorithm">From the Relay spec</a>:</p><blockquote>hasPreviousPage will be false if the client is not paginating with last, or if the client is paginating with last, and the server has determined that the client has reached the end of the set of edges defined by their cursors.</blockquote><p>Because of this behaviour, Relay doesn&#39;t allow for bi-directional pagination (as described above) out of the box but only for a uni-directional one (e.g. just paginating in one direction e.g. when scrolling down a news-feed). There are many inactive as well as ongoing <a href="https://github.com/facebook/relay/issues/540">discussions</a> on various Github repositories about this issue. Official maintainers voiced that Facebook chose this default behaviour due to the underlying systems being responsible for delivering the content having problems-by-design with constantly answering the questions if there are both previous &amp; next pages at the same time (which makes sense if you assume they are using a DynamoDB- or Cassandra-like datastore under the hood).</p><h3>How to implement bi-directional pagination Relay-style</h3><p>The obvious and maybe easiest solution in many cases would be to simply implement an offset-based navigation which is perfect for bi-directional use-cases.</p><p>However many frameworks might not support this out of the box and if you are also trying to spend as much of your time on user-facing problems like we do here at Dubsmash you might want to stick with the default Relay-style behaviour. Nevertheless there are multiple ways you can hack bi-directional pagination, one of which I want to outline here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29a98315e98ec892fa09d5c3767767bc/href">https://medium.com/media/29a98315e98ec892fa09d5c3767767bc/href</a></iframe><p>The overall idea is to build up a stack of cursors that you already used while navigating forwards through the pages. Every time a user is going forwards a page (e.g. pressing a “Next Page” button or similar) you push the current endCursor<em> </em>onto the cursor stack. Whenever the user chooses to go backwards again (e.g. pressing a “Previous Page” button or similar) you pop the last cursor from the stack (without actually using it) and then use the top-most cursor on the stack to fetch the page again.</p><p>If you are using a framework like <a href="http://dev.apollodata.com/">Apollo</a> this operation will be blazing fast since it automatically deploys advanced caching mechanisms making sure you don&#39;t even need to do another roundtrip to the server unless you specify a hasPreviousPage forcing it to do so.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b523c919c96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/bi-directional-pagination-using-graphql-relay-b523c919c96">Bi-directional pagination using GraphQL &amp; Relay</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple email validation service using AWS Lambda & Zappa]]></title>
            <link>https://medium.com/dubsmash-engineering/simple-email-validation-service-using-aws-lambda-zappa-3572e940ceee?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3572e940ceee</guid>
            <category><![CDATA[email]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Tue, 04 Jul 2017 16:29:41 GMT</pubDate>
            <atom:updated>2017-07-04T16:29:41.336Z</atom:updated>
            <content:encoded><![CDATA[<p>Here at Dubsmash, we are strong believers that technology should always be used to provide the product itself with a solid technical basis in order to enable rapid product development while providing an awesome user experience to our users. <a href="https://martinfowler.com/articles/serverless.html">Serverless architectures</a> (as in no permanent infrastructure) helps us quickly implement &amp; deploy our code to the cloud so we can keep improving the user experience. Frameworks like Zappa which are building on top of this principle provide enough tooling to enable us to bootstrap a complete service &amp; make it ready for production with minimal investment of time.</p><p><em>Disclaimer: This solution does not claim to be the most performant or complete but is simply a proof-of-concept to show-case the usage of Lambda &amp; Zappa.</em></p><p>The overall idea to validate email addresses is to use the targets SMTP (Simple Mail Transfer Protocol) server directly to verify it actually does exist. In order to do so we are going to build a small micro-service using <a href="http://flask.pocoo.org/">Flask</a> which will be deployed on AWS Lambda using Zappa that will provide an endpoint expecting an email address as input and returning a status code of either 400 or 200 depending on whether the email provided as input is valid or not.</p><ul><li><strong>Step 1:</strong> Get the MX records of the targeted email domain. We will be using the first one of usually at least 2 for the sake of this exercise. If one wants a complete solution the probability of one of the provided servers being unreachable should be handled more gracefully and a fallback to the remaining ones should be implemented.</li><li><strong>Step 2:</strong> Open up a connection to the SMTP server using the address provided in the MX record.</li><li><strong>Step 3</strong>: Start the SMTP flow by sending the <em>HELO</em> header followed by the mandatory <em>MAIL FROM</em> header. Usually, one would send the senders email address in this header but since we are not going to actually try sending a real email we will just use the email address to be validated.</li><li><strong>Step 4</strong>: Send the <em>RCPT</em> header to mark the email to be validated as the wanted receiver of the email you are trying to send.</li><li><strong>Step 5</strong>: Check the response of the <em>RCPT</em> message. If the status is 250 the SMTP server knows the email address we are trying to validate, if it&#39;s something else the email doesn&#39;t exist. In this case the error message will contain more information on why the <em>RCPT</em> command failed but we will just interpret every failure in general as “Email not existing” for now.</li></ul><p>The complete code should look somewhat like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/848c13112204f2c1d3827549f3947545/href">https://medium.com/media/848c13112204f2c1d3827549f3947545/href</a></iframe><p>As mentioned already, this post does not claim to provide a full production ready solution to run email validation in your application but leaves a lot of space for further improvements. A few improvements one could do are adding caching using <a href="https://memcached.org/">memcached</a> or <a href="https://redis.io/">Redis</a> for both SMTP responses &amp; MX lookups as well as adding more heuristical approaches before even doing the MX lookup in the beginning by testing for common typos in emails like <em>gmail.con</em> or <em>gmaik.com (which we at Dubsmash have found happening as surprisingly often on mobile!).</em></p><p>Serverless architectures provide product teams an easy &amp; maintainable way to quickly iterate on small services while still making sure to stay on-budget. <a href="https://github.com/dubsmash/dubsmash-email-validation">This complete project can be found on Github</a>.</p><p><em>There are many ways to improve the proposed setup and expand it to other use-cases. If you know them we probably would love to work with you: </em><a href="http://dubsmash.com/jobs"><em>Jobs @ Dubsmash</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3572e940ceee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/simple-email-validation-service-using-aws-lambda-zappa-3572e940ceee">Simple email validation service using AWS Lambda &amp; Zappa</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing real-time analytics using AWS Kinesis & Lambda]]></title>
            <link>https://medium.com/dubsmash-engineering/implementing-real-time-analytics-using-aws-kinesis-lambda-1ea56f10e473?source=rss-808221e7725d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ea56f10e473</guid>
            <category><![CDATA[aws-kinesis]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Tim Specht]]></dc:creator>
            <pubDate>Sat, 08 Apr 2017 14:59:13 GMT</pubDate>
            <atom:updated>2017-04-08T14:59:13.557Z</atom:updated>
            <content:encoded><![CDATA[<p>At Dubsmash we are running a global video platform with millions of users every day. To get proper insights into our user behaviour and how new &amp; old features impact our platform we are generating an increasing amount of analytics data going into the hundreds of gigabyte a day. <a href="https://aws.amazon.com/kinesis/">AWS Kinesis</a> has proven to be a reliable source for ingesting those volumes into one single, distributed stream at any scale for us in the past and has been the core entry point into our data pipeline.</p><h3>Background</h3><p>Previously we were running hand-written Python workers using <a href="https://github.com/boto/boto3">Boto</a> to read data from the stream and prepare it for further processing. With ever increasing amounts of traffic, and since this code had it&#39;s quirk due to being written quite some time ago when Kinesis streams became generally available, we set out to rebuild this part of our analytics. The biggest drawbacks of our legacy implementation was the need to maintain the shard iterator logic enforced by Kinesis, storing them somewhere and making sure that we were not losing any records while processing the data.</p><p>Dubsmash is already using <a href="https://aws.amazon.com/lambda">AWS Lambda</a> for a variety of tasks and has found it&#39;s an easy way to scale operations to unlimited amounts of parallelisms on a trigger-like basis. Given the requirements of being able to handle vast amounts of data while still being as real-time as possible, with keeping costs at bay, Lambda sounded like the perfect tool for the task.</p><p>The current version of our analytics pipeline consists of 3 different steps:</p><ol><li>Clients send events as JSON blobs to one central Kinesis stream using the platform-native AWS SDKs.</li><li>The Kinesis stream is automatically polled by a Lambda function doing simple format-validation and parsing on every message received from the stream and writes the messages to an <a href="https://aws.amazon.com/sqs">AWS SQS</a> queue.</li><li>Workers pull messages from SQS and process them further on a per-event-type basis.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*ER5w5rTALPlvAGYoHCeu7A.png" /></figure><h3>Wiring up Lambda to Kinesis</h3><p>Given that you already have a Kinesis stream set up which receives data of any kind from producers of your choice you first need to create the Lambda function and configure a trigger from Kinesis.</p><p>To do so go to the <a href="http://console.aws.amazon.com/lambda/home">AWS Lambda management console</a> and click on “Create a Lambda function”. You are asked to select from multiple pre-defined templates to bootstrap your function but for the sakes of the exercise you want to select “Blank Function” for now. Next, you will be asked to set up a trigger for your function. The trigger defines what causes the Lambda function to execute automatically and, at this point, you can specify various services as sources. Now that you’ve specified your services, choose “Kinesis” and the Kinesis stream that contains your data.</p><h3>Define the trigger</h3><p>When configuring Kinesis as the trigger for Lambda you have two key parameters to specify: The batch size and the starting position.</p><p>The batch size defines how many records from the stream are allowed as the maximum amount your function is executed with. This does not mean that AWS will wait for the amount you specify to accumulate before executing the function but only that within a given time window of 200ms between the polls to Kinesis your function will not be called with more than &lt;batch size&gt; records at once. When playing around with this value at Dubsmash it turned out that values between 100 and 1000 seem to be sane ones but this strongly depends on your specific use-case. Keep in mind: The more records your function receives as input the longer it will need to execute and the more RAM it will need which will lead to higher costs on your end. For more information check out <a href="https://aws.amazon.com/lambda/pricing/">this documentation on AWS Lambda pricing</a>.</p><p>The starting position is used to define where in your stream Lambda should start reading records once you enable the trigger. The three different possible values are <em>TRIM_HORIZON</em>, <em>LATEST</em> and <em>AT_TIMESTAMP</em>. <em>TRIM_HORIZON</em> is the one you will be most likely to use since it defines the starting point at the oldest record in the stream first and then continue towards the newer ones. Once the function has caught up to the end (or beginning) of your stream it will continue to make sure that every new record is still being read.. <em>LATEST</em> is doing the inverse e.g. starting at the newest record and discarding anything before. <em>AT_TIMESTAMP</em> starts at a given timestamp within the stream.</p><h3>Adding the code</h3><p>Once you selected appropriate values make sure to enable your trigger by checking the “Enabled” checkbox and hit next. You&#39;ll be asked to specify a name of your choice for your function and the language environment you want to use for your code. I&#39;ll use Python from here on and skip explanations on uploading pre-packaged environments including dependencies in virtualenvs and will focus on in-line code editing. If you are interested in more sophisticated Lambda setups feel free to check out <a href="http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html">this</a> as a starting point.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf093b0f1c533acc75264f08fb1c8417/href">https://medium.com/media/cf093b0f1c533acc75264f08fb1c8417/href</a></iframe><p>As you can see above, the <em>lambda_handler</em> function is the entry point called by AWS for our function. It receives the event created by the trigger we created earlier which includes the Kinesis records itself as well as the context which contains basic information about the function itself and the context it runs in (you can find more information on the context <a href="http://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html">here</a>). As you can see above we are checking a custom <em>async</em> parameter in the event payload. This is used to have the function trigger itself with a sub-set of the payload. The reasoning for this is that Lambda polls every Kinesis shard 5 times a second (<a href="http://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html">due to Kinesis limits</a>) e.g. every 200ms. If the function is too slow in processing the data it will not be able to keep up with the stream and will start dropping data at some point (more on that later). To solve this we simply split up the payload into smaller chunks of &lt; 130kb (due to <a href="http://docs.aws.amazon.com/lambda/latest/dg/limits.html">Lambda payload size limits</a>) and invoke the function again asynchronously to multiplex the actual processing. To avoid an endless loop we add the additional <em>async</em> flag to the context to let the instance of the function which we invoke know that it should directly process the data.</p><p>The <em>process_records</em> function is simply aggregating records together into batches before sending them to SQS to make sure we are using the smallest amount of requests to SQS to keep costs down (more information on pricing can be found <a href="https://aws.amazon.com/sqs/pricing/">here</a>).</p><h3>Further processing</h3><p>With the data having made its way all the way down to SQS we can now process the records further. The <a href="http://boto3.readthedocs.io/en/latest/guide/sqs.html#processing-messages">boto3 documentation</a> covers this quite extensively so I&#39;ll omit this part for now.</p><h3>Pitfalls</h3><p>While Kinesis and Lambda make a good match when used in conjunction, there are a couple of pitfalls we encountered while implementing this architecture which I&#39;d like to point out to avoid others the same grievance:</p><ol><li><strong>Batch size:</strong> Be aware that once a trigger is created you cannot edit the batch size of said trigger. The only possibility to do so is to delete the trigger and create a new one with a different batch size once your function is created. While this may seem like nothing huge and indeed is very little work it has quite some implications: If you use the TRIM_HORIZON configuration when creating the trigger for your function every time you create a new trigger your function will start reading at the very beginning of the stream since the underlying shard-iterator (more information on shard iterators can be found <a href="http://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html#kinesis-using-sdk-java-get-data-shard-iterators">here</a>) is a fresh one. This means your function will automatically start reading at the oldest available record again and hence will re-process all the items it maybe already processed until it reached the beginning of the stream again. Make sure you have proper duplicate filtering in place and prepare for this process to take some time to catch up again depending on the size of your stream!</li><li><strong>IteratorAge:</strong> One of the key metrics we found useful was the <em>IteratorAge</em> you can find in either the Kinesis or the Lambda part of your CloudWatch console. The <em>IteratorAge</em> graphs how big the time difference is between where your function is currently reading in the stream and the actual beginning of the stream. As a small example: If you <em>IteratorAge</em> metric value is 40 million milliseconds this means that the latest record processed by your Lambda function is 11.1 hours old. If your IteratorAge is too high that means your function is not able to keep up with the stream and will start dropping data as soon as it reaches the retention period of your stream! When you want your pipeline to be real-time you should try to keep this value under an average of 1000 milliseconds.</li><li><strong>How Lambda polls the stream</strong>: As mentioned earlier already Lambda is polling the Kinesis stream on a per-shard basis 5 times per second which is the maximum amount of reads Kinesis allows. This means your function is only executed every 200ms and you only have one invocation per 200ms per shard which is limiting your parallelism as soon as you start to hit a certain data volume! The reasoning for this is that Lambda has to persist the shard iterator it&#39;s using behind the scenes to know where it stopped reading last in the stream and Kinesis only allows one iterator per shard. Luckily Kinesis allows us to create multiple shards (which we need anyway due to the amount of data coming into our stream) and hence provides ways to spread out also the amount of Lambda functions reading from the stream. You can read more about Kinesis sharding <a href="http://docs.aws.amazon.com/streams/latest/dev/key-concepts.html">here</a>.</li><li><strong>Sharding:</strong> While we already had this properly set up I find it notable still: Make sure to have enough shards available in your stream to handle the traffic you expect to flow in as well as the data being read. Even more important make sure that when ingesting the data into your Kinesis stream you are using a proper partition key. This partition key is used by Kinesis to hash it and then distribute it to the shards you have allocated to your stream based on the calculated hash. If you would be using for example always “1” as the partition key all your records would go to one single shard which would soon collapse under the traffic while the rest of the allocated shards stay unused. You can read more about proper partition keys <a href="http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html">here</a>.</li></ol><h3>Further improvements</h3><p>The architecture described above does simply provides us with an easy way to have AWS handle the complexity of juggling multiple shard iterators and reading data from Kinesis and doesn&#39;t contain any actual logic. To further improve our setup and rely more on serverless processing using Lambda the following things are worth considering:</p><ul><li>Have Lambda actually validate data schemas and the data itself before passing it down further down the pipeline</li><li>Have Lambda do duplicate detection itself by reading/writing event IDs to and from a Redis instance or something similar</li><li>Using different Lambda functions to implement further steps in the pipeline like pre-aggregating data or writing it to data stores used for further offline processing and querying</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ea56f10e473" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dubsmash-engineering/implementing-real-time-analytics-using-aws-kinesis-lambda-1ea56f10e473">Implementing real-time analytics using AWS Kinesis &amp; Lambda</a> was originally published in <a href="https://medium.com/dubsmash-engineering">Dubsmash Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>