<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Signiant Engineering - Medium]]></title>
        <description><![CDATA[Interesting projects from the Signiant engineering team - Medium]]></description>
        <link>https://medium.com/signiant-engineering?source=rss----cc5a48dd2b94---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Signiant Engineering - Medium</title>
            <link>https://medium.com/signiant-engineering?source=rss----cc5a48dd2b94---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:21:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/signiant-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using DynamoDB transactions to model relational data]]></title>
            <link>https://medium.com/signiant-engineering/using-dynamodb-transactions-to-model-relational-data-c13619ca58ae?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/c13619ca58ae</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Peter Hodgkinson]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 16:13:36 GMT</pubDate>
            <atom:updated>2019-09-06T13:37:12.214Z</atom:updated>
            <content:encoded><![CDATA[<p>At Signiant we use AWS’s <a href="https://aws.amazon.com/dynamodb">DynamoDB</a> extensively for storing our data. We like it because it provides scalability and performance while being almost completely hands-off from an operational perspective.</p><p>AWS <a href="https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-transactions/">recently announced</a> support for transactions in DynamoDB. Transactions guarantee that a set of database operations are performed as an atomic unit, and are <a href="https://en.wikipedia.org/wiki/ACID_(computer_science)">ACID</a> compliant. In the case of an error during any part of the transaction, all operations will be rolled back, helping to guarantee data integrity and consistency across many rows or tables.</p><p>This is particularly useful for modelling relational data, where relationships between entities are written across multiple records and all need to be present to maintain integrity.</p><p>We’re going to run through an example of how one might leverage transactions when writing relational data in DynamoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*fenT4Ze1t2PrdeXT.jpg" /></figure><h3>Sample Relational Data Scenario</h3><p>Let’s take a simplified example of flights between airports. In this scenario we have three related entities:</p><ol><li>Airports</li><li>The routes or connections between airports</li><li>Airline companies that operate routes</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*zuvP3nH3ixbvvMWIgcdkaw.png" /><figcaption>Sample routes between three airports</figcaption></figure><p>When modelling relational data in a NoSQL database like DynamoDB, it is important to think about how you will want to access the data before you start. In our case, we want to organize the data in a way to answer the following questions:</p><ol><li>What routes does a given company operate?</li><li>What routes enter or leave airport A, or connect airport A and airport B?</li></ol><p>Using techniques described in the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-relational-modeling.html">Best Practices for Modeling Relational Data with DynamoDB</a>, we will start with the following rows in our table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*hJac_wGyVPMOhrgJIDxvrw.png" /><figcaption>Sample Data</figcaption></figure><p>We can then use standard and efficient DynamoDB queries to retrieve the data for the access patterns we defined.</p><blockquote>What routes does Company-A operate?</blockquote><p>Use the Global Secondary Index partition key (GSI-PK) to query on <em>Company-A. </em>Extract the route from the PK attribute to get “Route-1” and “Route-2”.</p><blockquote>What routes connect LAX and JFK?</blockquote><p>Use the primary key to query where PK=<em>Airport-LAX</em>, SK <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html">BEGINS_WITH</a> <em>Airport-JFK. </em>Extract the route from the SK attribute to get “Route-1” .</p><h3>Adding a new route into the system</h3><p>Let’s say that “<em>Undercutter Air” </em>is introducing a new flight direct from LAX to LFH.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*8TLsnor7l-l7hkAfj5iUiA.png" /><figcaption>Newly added route</figcaption></figure><p>To add this route into our system we need to insert three new rows to our table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*GQv-I-3ijcMUWsUWVtkxug.png" /><figcaption>Newly Added Rows</figcaption></figure><p>Historically, your best option to add these three rows as an atomic unit was to perform a <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html">batch write</a>. Using the AWS JavaScript SDK, the code would look something like this:</p><pre><strong>const params </strong>= {<br>    <strong>RequestItems</strong>: {<br>        <strong>&#39;Table-1&#39;</strong>: [{<br>            <strong>PutRequest</strong>: {<br>                <strong>Item</strong>: {<br>                    <strong>PK</strong>: <strong>&#39;Route-3&#39;</strong>,<br>                    <strong>SK</strong>: <strong>&#39;Company-B&#39;</strong>,<br>                    <strong>From</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                    <strong>To</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>                }<br>            }<br>        }, {<br>            <strong>PutRequest</strong>: {<br>                <strong>Item</strong>: {<br>                    <strong>PK</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                    <strong>SK</strong>: <strong>&#39;Airport-LHR|Route-3&#39;</strong>,<br>                }<br>            }<br>        }, {<br>            <strong>PutRequest</strong>: {<br>                <strong>Item</strong>: {<br>                    <strong>PK</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>                    <strong>SK</strong>: <strong>&#39;Airport-LAX|Route-3&#39;</strong>,<br>                }<br>            }<br>        }]<br>    }<br>};<br><br><strong>const documentClient </strong>= <strong>new </strong>AWS.DynamoDB.DocumentClient();<br><br><strong>documentClient</strong>.batchWrite(<strong>params</strong>, <strong>function </strong>(err, data) {<br>    <strong>if </strong>(err) <strong>console</strong>.log(err);<br>    <strong>else console</strong>.log(data);<br>});</pre><p>While this would work a vast majority of the time, there are cases where a portion of the batch write could fail, leaving your data in an inconsistent state. You can, and should, follow the recommendations of retrying any unprocessed keys until they succeed. But for how long do you retry? What happens if your system shuts down while you are retrying? What happens if somebody tries to query the data when only a portion of the keys are written? There are no simple solutions to these problems.</p><h3>Using DynamoDB Transactions</h3><p>To write the same data as an atomic transaction, we can instead use the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html">TransactWrite</a> call in the JavaScript SDK. The code would look something like this:</p><pre><strong>const params </strong>= {<br>    <strong>TransactItems</strong>: [{<br>        <strong>Put</strong>: {<br>            <strong>TableName </strong>: <strong>&#39;Table-1&#39;</strong>,<br>            <strong>Item</strong>: {<br>                <strong>PK</strong>: <strong>&#39;Route-3&#39;</strong>,<br>                <strong>SK</strong>: <strong>&#39;Company-B&#39;</strong>,<br>                <strong>From</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                <strong>To</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>            }<br>        }<br>    }, {<br>        <strong>Put</strong>: {<br>            <strong>TableName </strong>: <strong>&#39;Table-1&#39;</strong>,<br>            <strong>Item</strong>: {<br>                <strong>PK</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                <strong>SK</strong>: <strong>&#39;Airport-LHR|Route-3&#39;</strong>,<br>            }<br>        }<br>    }, {<br>        <strong>Put</strong>: {<br>            <strong>TableName </strong>: <strong>&#39;Table-1&#39;</strong>,<br>            <strong>Item</strong>: {<br>                <strong>PK</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>                <strong>SK</strong>: <strong>&#39;Airport-LAX|Route-3&#39;</strong>,<br>            }<br>        }<br>    }]<br>};<br><br><strong>const documentClient </strong>= <strong>new </strong>AWS.DynamoDB.DocumentClient();<br><br><strong>documentClient</strong>.transactWrite(<strong>params</strong>, <strong>function</strong>(err, data) {<br>    <strong>if </strong>(err) <strong>console</strong>.log(err);<br>    <strong>else console</strong>.log(data);<br>});</pre><p>That’s it! It will succeed or fail as an atomic unit. No need for the added complexity of retries, or to worry about partial failures causing data integrity issues.</p><p>You can also perform transactions across tables, or perform deletes and updates in the same transaction, if your relational model requires it.</p><p><strong>Update: Note </strong><a href="https://aws.amazon.com/about-aws/whats-new/2019/06/amazon-dynamodb-now-supports-up-to-25-unique-items-and-4-mb-of-data-per-transactional-request/">as of June 2019</a> there is now an increased <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-dynamodb-transactions">limit of 25 operations</a> per transaction, so if you need to update more than 25 rows you may need to revert to batch writing or perform multiple transactions.</p><h3>References</h3><p>You can learn more about advanced design patterns in DynamoDB by watching <a href="https://www.youtube.com/watch?v=HaEPXoXVf2k">this great presentation from Re:Invent 2018</a>.</p><p>Learn more about modelling relational data from AWS best practices:</p><ul><li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-relational-modeling.html">Best practices for modeling relational data in DynamoDB</a></li><li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html">Best practices for managing many-to-many relationships in DynamoDB tables</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c13619ca58ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/using-dynamodb-transactions-to-model-relational-data-c13619ca58ae">Using DynamoDB transactions to model relational data</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The most interesting AWS service…in the world!]]></title>
            <link>https://medium.com/signiant-engineering/the-most-interesting-aws-service-in-the-world-3684d3fa8d4d?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/3684d3fa8d4d</guid>
            <category><![CDATA[amazon-mechanical-turk]]></category>
            <category><![CDATA[interesting]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dave North]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 20:17:59 GMT</pubDate>
            <atom:updated>2018-10-16T21:34:39.555Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*WODtNk5QRgupCLqPhVe-Jg.jpeg" /><figcaption>Mechanical Turk</figcaption></figure><p><em>Edit: As was pointed out in the comments, Turk isn’t strictly an AWS service but rather an Amazon service. As it’s in the AWS SDK as a service though, it is related to AWS. I have not edited the text of this article but offer this correction here</em></p><p>Anyone up on their AWS Jeopardy knows that the first AWS service was SQS. But what’s the most interesting AWS service? DynamoDB? Lex? Lambda would have to be right up there (I recall vividly the gasp from the audience at Re:Invent in 2015 when it launched). I was recently reminded of what I think is the most interesting and unique service that I’ve used though and thought I’d write about how we used it. The service is <a href="https://docs.aws.amazon.com/mturk/index.html#lang/en_us">Mechanical Turk</a>.</p><p>I was reminded about this service recently while listening to a lunch and learn from a colleague talking about all the great information he’d learnt at the <a href="https://aws.amazon.com/summits/toronto/">AWS Summit in Toronto</a>. One area of interest for my colleague is machine learning and in the AWS machine learning stack, there’s a small box for something called Mechanical Turk alongside the more widely known Sagemaker.</p><h3>What is Mechanical Turk?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*buQsh_i9ZrvH58w8srsDyQ.jpeg" /><figcaption>The most interesting service…in the world!</figcaption></figure><p>Turk is a scalable “army” of human workers that are available on-demand. The basic idea is there are 2 type of users — requesters and workers. Requesters post small units of work for…well, workers to work on. These are generally simple tasks that pay out typically pennies. A common example of a work item would be image tagging for example.</p><h3>How did Signiant use it?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*7fm7yqiDLsik9Dx4.jpg" /></figure><p>One of the perennial problems in software is always getting enough beta and test feedback on a new release or feature before it’s unleashed to the wild. Lots of people show interest in a beta release but few actually provide meaningful (or any!) feedback.</p><p>Some years back (circa 2012), we were working on a new feature for our SAAS product and before we generally released it, we wanted to get the feedback and <em>real world</em> experience from as many people as possible — around 1000 if we could do it. After looking around for a good solution for this, I came across <a href="https://www.mturk.com/mturk/welcome">Amazon Mechanical Turk</a> which sounded promising. While it wasn’t typically used for the case we were looking at, it did provide access to a wide variety of workers using various end-user platforms and browsers.</p><p>In our case, we needed someone to use a web portal to upload a file and answer some basic questions:</p><ul><li>What OS are you running?</li><li>What browser are you using?</li><li>What available bandwidth do you have?</li><li>etc.</li></ul><p>We posted a work item (called a <strong>HIT</strong> or <em>Human Intelligence Task</em> in Turk speak) and waited to see if we could get anyone to bid and work on it. I asked for 10 workers for the first test with a price of 50c per HIT with the restriction that all workers had to be unique (ie. we didn’t want all work completed by the same worker). We got all 10 — 9 of them could complete the task, 1 could not. The feedback was fantastic.</p><p>Based on this small experiment we were able to scale this out and get feedback from more workers. Our HIT attracted some good quality feedback and because it paid well and was outside the usual “tag this image” task, our HIT was rated highly.</p><p>I think Mechanical Turk is generically very useful for spot testing and general usability quick tests. The only piece that thwarts more adoption is your HITs cannot have the workers install any software so that may limit some use cases. Web based tasks are fine though.</p><p>Over the years, I’ve used many fantastic AWS services (Lambda still amazes me) but Mechanical Turk is still the most interesting (and out there!) service I’ve come across.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3684d3fa8d4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/the-most-interesting-aws-service-in-the-world-3684d3fa8d4d">The most interesting AWS service…in the world!</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consolidating AWS Load Balancers using host-based routing]]></title>
            <link>https://medium.com/signiant-engineering/consolidating-aws-load-balancers-using-host-based-routing-212f5949fca7?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/212f5949fca7</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[application-load-balancer]]></category>
            <dc:creator><![CDATA[Dave North]]></dc:creator>
            <pubDate>Wed, 10 Oct 2018 19:41:14 GMT</pubDate>
            <atom:updated>2018-10-10T19:41:14.048Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*cTNWrcqyA2EtyBBkFjkuYw.jpeg" /></figure><p>We’ve been using AWS ELBs for many years now across a variety of our cloud services. As someone from “the old times” who has configured hardware LBs before, they still seem like wizardry! Spin up an infinitely scalable load balancer via boto/CLI/console for only just over $20/month?! What a deal!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*k8n3WggM-azaneIM.jpg" /></figure><p>The $deal$ can end up being quite a chunk of the bottom line if you use enough ELBs though. Several years ago, we started decomposing one of our monoliths to a microservice pattern using ECS containers. For service discovery and scalability, we used an ELB per service with a Route53 endpoint for the load balancer. Predictable and scalable and with not too many services to start with, not overly expensive.</p><p>However, as the number of services increased, so the did the cost. When Application Load Balancers (ALBs) were released, we switched to these so we could use dynamic port mapping for our ECS containers. We continued to spin up a new load balancer per service using our “service-in-a-box” cloudformation pattern which works very well for deploying a service anywhere.</p><p>One thing to inject here is the <em>environment cost multiplier</em> effect we have here. For reasons not important to this article, we have many dev — test — prod environments that need to run these services. 9 in total. So while each load balancer is <em>only</em> approx. $20/month, it actually becomes $180/month with all the environments. And that’s on top of any other costs to run the actual service (ECS is <em>VERY</em> cost effective for running containerized services when using EC2 instances for the cluster; Fargate, not so much).</p><p>So we have a great pattern for deploying services but the cost is increasing linearly as we add more services. Alarm bells started to ring….</p><h3>A Better Way</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*YCN25kgnssuESpQu.jpg" /></figure><p>After moving the services to Application load balancers (ALBs), we looked at using <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/tutorial-load-balancer-routing.html">path based routing</a>. In this model, rather than your service just receiving requests on the root (/) endpoint, you can define an endpoint on the ALB and have it direct traffic to the appropriate service. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/0*uvLO2cyBFckVEkAX.png" /></figure><p>In our case though, the problem is each service would have had to have its code modified to handle the new incoming URI. It wasn’t transparent to the developers. So we put this on hold for a while.</p><p>In April 2017, AWS came up with <a href="https://aws.amazon.com/blogs/aws/new-host-based-routing-support-for-aws-application-load-balancers/">host-based routing</a> for ALBs. This is similar to path-based routing but instead of using a path suffix, the host header is examined and requests are routed accordingly. This seemed a lot more promising as it should be totally transparent to the underlying service.</p><p>And it was!</p><p>We ended up moving to this model and cutting our load balancers required from one per service to one per environment. ie. if we had 30 services across 9 environments, we needed 270 load balancers. Under the new model, we would need 9. A 97% reduction!</p><p>The rest of this article will talk about how we implemented this and one of the bigger, unexpected problems we had to solve.</p><h3>The Path of Change</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*9Dcc5hMR_5mMEWak.jpg" /></figure><p>The actual work we had to do consisted of 3 parts:</p><ul><li>Create a shared load balancer in each of the environments</li><li>Modify our existing pipeline scripts to be able to compute the next ALB rule priority (this was actually the biggest pain)</li><li>Re-package our existing services to use the common ALB</li></ul><h3>Creating the Shared ALB</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/0*-nArAuLFR80ooPMN.jpg" /></figure><p>This is done via a simple cloudformation template (a generic version of which is available on <a href="https://github.com/dnorth98/aws-alb-consolidation/blob/master/common-alb/common-alb.yaml">GitHub</a>). There’s nothing terribly challenging about this but there are 2 things of note:</p><ul><li>Not all of our services are in the same DNS zone. Thankfully, ALBs allow attaching up to 25 SSL certs to a single listener. In our case we only needed 5 so we prompt for the cert ARNs and attach to the listener if supplied using conditions in the Cloudformation template:</li></ul><pre>Conditions:<br>    AddLBSSLCert2: !Not [ !Equals [ !Ref LBSSLCERT2, &quot;&quot; ] ]<br>    AddLBSSLCert3: !Not [ !Equals [ !Ref LBSSLCERT3, &quot;&quot; ] ]<br>    AddLBSSLCert4: !Not [ !Equals [ !Ref LBSSLCERT4, &quot;&quot; ] ]<br>    AddLBSSLCert5: !Not [ !Equals [ !Ref LBSSLCERT5, &quot;&quot; ] ]</pre><ul><li>At the end of the template, we use <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-exports.html">Cloudformation exports</a> to export the information about the load balancer (ARN, DNS name, etc.). Our individual services then use these as <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html">Cloudformation imports</a> into their own Cloudformation templates to attach to the correct ALB:</li></ul><pre>Outputs:<br>    ALBARN:<br>        Value: !Ref MicroserviceLoadBalancer<br>        Export:<br>          Name: !Join [&quot;-&quot;, [ !Ref SIGNIANTENV, &quot;microservice-alb&quot;, !Ref ALBNUMBER, &quot;arn&quot; ] ]<br>    ALBLISTENERARN:<br>        Value: !Ref ALBListener<br>        Export:<br>          Name: !Join [&quot;-&quot;, [ !Ref SIGNIANTENV, &quot;microservice-alb&quot;, !Ref ALBNUMBER, &quot;listenerarn&quot; ] ]<br>    ALBZONEID:<br>        Value: !GetAtt MicroserviceLoadBalancer.CanonicalHostedZoneID<br>        Export:<br>          Name: !Join [&quot;-&quot;, [ !Ref SIGNIANTENV, &quot;microservice-alb&quot;, !Ref ALBNUMBER, &quot;zoneid&quot; ] ]<br>    ALBDNSNAME:<br>        Value: !GetAtt MicroserviceLoadBalancer.DNSName<br>        Export:<br>          Name: !Join [&quot;-&quot;, [ !Ref SIGNIANTENV, &quot;microservice-alb&quot;, !Ref ALBNUMBER, &quot;dnsname&quot; ] ]</pre><p>If we need more than one ALB per region/environment, we’ll create multiple stacks from the same template and just increment the “load balancer number” parameter to give everything a unique ID.</p><h3>Modifying the Pipeline Scripts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*2u7E8D6mPVVCotS6.jpg" /></figure><p>In the “nothing is ever easy” camp, this was the unexpected piece of work we had to do. Each rule added to a listener needs a unique priority — even if with host-based routing this doesn’t make a lot of sense. Because the priority has to be unique for each service, we needed a way to manage the priorities for each host route we add. After engaging AWS support to confirm that there was no built in way to do this (something similar to how ECS host ports are dynamically assigned would be nice!), we realized we’d have to produce our own solution.</p><p>What we ended up with was a small routine added to our deployment pipeline which computes the next rule priority at promotion time (ie. use the ALB as the master list of priorities). This means that the same service could have a different priority in different environments but it really doesn’t matter with host-based routing as the hostname either matches or it doesn’t. Using some bash and the AWS CLI, we found an algorithm that worked for us (even handling the case where we have multiple rules for the same service in some environments). You can see the code snippet from our release pipeline in <a href="https://github.com/dnorth98/aws-alb-consolidation/blob/master/alb-rule-priority-computer/alb_priorities.sh">GitHub</a> but basically it’s:</p><ul><li>See if we already have a rule for “this” hostname. If we do, get the priority and add it to the Cloudformation parameters</li><li>If we don’t, get the max rule priority and add 1 to it</li><li>If we need to add a “regional record”, compute the next priority for this record</li></ul><h3>Re-Package the Existing Services</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_AT8_ydTDQcX4tGZ.jpeg" /></figure><p>I’ve included our standard “service in a box” Cloudformation template in the GitHub repo. There’s 2 files there — the actual cloudformation template and a second YAML file containing the parameters that are passed. This second YAML file of parameters is filled in at build time and generated per environment we need to release to. The priorities for the service are computed by the pipeline script above and added into the parameters YAML for each environment. Specifically, these values:</p><pre>LISTENERRULEMAINPRIORITY: &quot;LISTENERRULEMAINPRIORITY_COMPUTED_AT_PROMO_TIME&quot;<br>LISTENERRULEREGIONALPRIORITY: &quot;LISTENERRULEREGIONALPRIORITY_COMPUTED_AT_PROMO_TIME&quot;</pre><p>These are then passed into the Cloudformation stack as these values:</p><pre>Parameters:<br>    LISTENERRULEMAINPRIORITY:<br>        Description: Priority to give the ALB listener rule for primary DNS<br>        Type: String<br>        Default: &quot;COMPUTED_AT_PROMO_TIME&quot;<br>    LISTENERRULEREGIONALPRIORITY:<br>        Description: Priority to give the ALB listener rule for regional DNS<br>        Type: String<br>        Default: &quot;COMPUTED_AT_PROMO_TIME&quot;</pre><p>Just a small note here on the service target group. This was an interesting nuance that showed up in testing. The target group is defined in the Cloudformation template as:</p><pre>ServiceTargetGroup:<br>        Type: &quot;AWS::ElasticLoadBalancingV2::TargetGroup&quot;<br>        Properties:<br>            HealthCheckPath: &quot;/healthcheck&quot;<br>            HealthCheckProtocol: HTTP<br>            Port: 80<br>:<br>:</pre><p>In the original template for all our services, the logical name for this resource was just <em>TargetGroup</em>. However, by renaming it to <em>ServiceTargetGroup</em>, we were able to do an update to the service with no downtime since Cloudformation will create a brand new resource rather than replacing the existing resource (changing the target group on an ECS service requires a service replacement). By simply renaming the logical resource, we got a seamless upgrade “for free”</p><p>Now we get to the main event — the listener rule. The old model had services all creating their own ALB. In this new model, we just need a new <em>AWS::ElasticLoadBalancingV2::ListenerRule</em> object that is attached to our common ALB. It looks like this:</p><pre>ListenerRuleMain:<br>      Type: &quot;AWS::ElasticLoadBalancingV2::ListenerRule&quot;<br>      Properties:<br>        Actions:<br>        - Type: forward<br>          TargetGroupArn:<br>            Ref: ServiceTargetGroup<br>        Conditions:<br>          - Field: host-header<br>            Values:<br>              - !Join [ &quot;&quot;, [ !Ref DNSPREFIX, ., !Ref DNSZONE] ]<br>        ListenerArn:<br>          Fn::ImportValue:<br>            !Sub &quot;${SIGNIANTENV}-${ALB}-listenerarn&quot;<br>        Priority: !Ref LISTENERRULEMAINPRIORITY</pre><p>Note the use of the <strong><em>Fn::ImportValue</em></strong> function to use the exported values from the common ALB stack. This simplifies things a great deal because this service doesn’t need to be configured with the ARN of it’s ALB — we just use the logical name of the ALB we want to attach to and get its export from its Cloudformation stack</p><p>Finally, we can add a Route53 DNS entry to point to the common ALB for this service name. The host-routing then takes over and directs traffic to the correct service behind the ALB based on its hostname in the request. Again, we use the <strong><em>Fn::ImportValue</em></strong> function to get the correct Route53 DNS Zone ID of the common ALB we want to use as well as the name of the ALB.</p><pre>DNSSingleRecord:<br>        Condition: &quot;CreateRegionalDnsEntry&quot;<br>        Type: &quot;AWS::Route53::RecordSet&quot;<br>        Properties:<br>            AliasTarget:<br>                HostedZoneId:<br>                  Fn::ImportValue:<br>                    !Sub &quot;${SIGNIANTENV}-${ALB}-zoneid&quot;<br>                DNSName:<br>                  Fn::ImportValue:<br>                    !Sub &quot;${SIGNIANTENV}-${ALB}-dnsname&quot;<br>            HostedZoneName: !Join [ &quot;&quot;, [ !Ref DNSZONE, .] ]<br>            Name: !Join [ &quot;&quot;, [ !Ref DNSREGIONPREFIX, ., !Ref DNSZONE, .] ]<br>            Type: A</pre><p>There’s obviously a lot more automation around the entire deployment process which we’ve <a href="https://medium.com/signiant-engineering/micro-service-deployment-process-using-jenkins-cloudformation-ecs-and-docker-121a0b8b10bf">written about before</a> but this post covers the basics of how we were able to use a great ALB feature to shrink our costs by 97% and simplify our overall infrastructure.</p><h3>Referenced Project</h3><p><a href="https://github.com/dnorth98/aws-alb-consolidation">https://github.com/dnorth98/aws-alb-consolidation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=212f5949fca7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/consolidating-aws-load-balancers-using-host-based-routing-212f5949fca7">Consolidating AWS Load Balancers using host-based routing</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Aggregation with DynamoDB Streams]]></title>
            <link>https://medium.com/signiant-engineering/real-time-aggregation-with-dynamodb-streams-f93547cfb244?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/f93547cfb244</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <dc:creator><![CDATA[Peter Hodgkinson]]></dc:creator>
            <pubDate>Mon, 11 Jun 2018 16:58:25 GMT</pubDate>
            <atom:updated>2019-12-13T14:46:53.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gVDQ4sEfoKssi5km" /><figcaption>Photo by <a href="https://unsplash.com/@hharritt?utm_source=medium&amp;utm_medium=referral">Hunter Harritt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At Signiant we help our customers move their data quickly. Some of our customers transfer <em>a lot</em> of data. Terabytes upon terabytes, every month.</p><p>We also strive to give our customers insight into how they are using our product, and feedback on how much data they are moving.</p><p>Building a system to meet these two requirements leads to a typical problem in data-intensive applications:</p><blockquote>How do you collect and write a ton of data, but also provide an optimal way to read that same data? Can you build this system to be scalable? Resilient to errors? Is it easy to implement and operate?</blockquote><p>A typical solution to this problem would be to write a batch process for combining this mass of data into aggregated rows. This approach has a few inherent problems:</p><ul><li>You need to operate and monitor a fleet of servers to perform the batch operations. Often this comes in the form of a Hadoop cluster.</li><li>You need to schedule the batch process to occur at some future time. But what happens if you want to query the data before that time?</li><li>What happens when something goes wrong with the batch process? Do you know how to resume from the failure point? Have you lost any data? What does it mean for your application if the previous batch didn’t succeed?</li></ul><p>Is there a better way? Can you produce aggregated data in real-time<em>,</em> in a scalable way, without having to manage servers?</p><h3>Using DynamoDB Streams</h3><p>At Signiant we use AWS’s <a href="https://aws.amazon.com/dynamodb">DynamoDB</a> extensively for storing our data. We like it because it provides scalability and performance while being almost completely hands-off from an operational perspective.</p><p><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html">DynamoDB Streams</a> is a feature of DynamoDB that can send a series of database events to a downstream consumer. This consumer can be an application you write and manage yourself, or an AWS <a href="https://aws.amazon.com/lambda">Lambda</a> function you write and allow AWS to manage and trigger. Depending on the operation that was performed on your source table, your application will receive a corresponding INSERT, MODIFY, or REMOVE event. The event will also include a snapshot of the data contained in the database row before and after it was changed.</p><p>For example, if a new row gets written to your source table, the downstream application will receive an INSERT event that will look something like this:</p><pre>{  <br>   <strong>&quot;Records&quot;</strong>:[  <br>      {  <br>         <strong>&quot;eventID&quot;</strong>:&quot;7de3041dd709b024af6f29e4fa13d34c&quot;,<br>         <strong>&quot;eventName&quot;</strong>:&quot;INSERT&quot;,<br>         <strong>&quot;eventVersion&quot;</strong>:&quot;1.1&quot;,<br>         <strong>&quot;eventSource&quot;</strong>:&quot;aws:dynamodb&quot;,<br>         <strong>&quot;awsRegion&quot;</strong>:&quot;us-east-1&quot;,<br>         <strong>&quot;dynamodb&quot;</strong>:{  <br>            <strong>&quot;ApproximateCreationDateTime&quot;</strong>:1479499740,<br>            <strong>&quot;Keys&quot;</strong>:{  <br>               <strong>&quot;Timestamp&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;2016-11-18:12:09:36&quot;<br>               },<br>               <strong>&quot;Username&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;John Doe&quot;<br>               }<br>            },<br>            <strong>&quot;NewImage&quot;</strong>:{  <br>               <strong>&quot;Timestamp&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;2016-11-18:12:09:36&quot;<br>               },<br>               <strong>&quot;Message&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;This is a bark from the Woofer social network&quot;<br>               },<br>               <strong>&quot;Username&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;John Doe&quot;<br>               }<br>            },<br>            <strong>&quot;SequenceNumber&quot;</strong>:&quot;13021600000000001596893679&quot;,<br>            <strong>&quot;SizeBytes&quot;</strong>:112,<br>            <strong>&quot;StreamViewType&quot;</strong>:&quot;NEW_IMAGE&quot;<br>         },<br>         <strong>&quot;eventSourceARN&quot;</strong>:&quot;arn:aws:dynamodb:us-east-1:123456789012:table/BarkTable/stream/2016-11-16T20:42:48.104&quot;<br>      }<br>   ]<br>}</pre><h3>Aggregating the data using AWS Lambda</h3><p><em>What if we use the data coming from these streams to produce aggregated data on-the-fly and leverage the power of AWS Lambda to scale-up seamlessly?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*K7ytEMpQCgVwgfhuc8ll3w.jpeg" /></figure><p>Presume we are writing records to a source DynamoDB table of the following schema:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dc9b046cd9451d3da5559d1f7f64ff8/href">https://medium.com/media/2dc9b046cd9451d3da5559d1f7f64ff8/href</a></iframe><p>If we want to produce a daily sum of all bytes transferred by a customer on a given day, our daily rollup table schema might look something like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39437a385eb4976c69ee38dacd9be693/href">https://medium.com/media/39437a385eb4976c69ee38dacd9be693/href</a></iframe><p>Given these two schemas, we want our system to take a set of rows from the source table that looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d834f3832ebccd2c198bfc34512b4bc5/href">https://medium.com/media/d834f3832ebccd2c198bfc34512b4bc5/href</a></iframe><p>And produce entries in the aggregated table that looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dec5c22bc2c4ead0d27ac921e466f11/href">https://medium.com/media/3dec5c22bc2c4ead0d27ac921e466f11/href</a></iframe><p>In the real world we write tens of thousands of rows into the source table per customer per day. It quickly becomes apparent that simply querying all the data from the source table and combining it on-demand is not going to be efficient. However querying a customer’s data from the daily aggregation table will be efficient for many years worth of data.</p><h4>The Lambda Function (Node 8.10)</h4><p><em>Note that the following assumes you have created the tables, enabled the DynamoDB stream with a Lambda trigger, and configured all the IAM policies correctly. You refer to </em><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.Tutorial.html"><em>this tutorial</em></a><em> for a quick overview of how to do all this.</em></p><pre><strong>const </strong>AWS = require(<strong>&#39;aws-sdk&#39;</strong>);<br><strong>const </strong>documentClient = <strong>new </strong>AWS.<strong>DynamoDB</strong>.<strong>DocumentClient</strong>({});<br><strong>async function </strong>handler(event, context, callback) {<br>    <strong>try </strong>{<br>        <em>//trim down to just &quot;INSERT&quot; events<br>        </em><strong>const </strong>insertRecords = event.<strong>Records</strong>.filter(<strong>record </strong>=&gt; record.eventName === <strong>&#39;INSERT&#39;</strong>);<br><em>// Unnmarshall records them to plain JSON objects<br>        </em><strong>const </strong>unmarshalledRecords = insertRecords.map(<strong>record </strong>=&gt;<br>            AWS.<strong>DynamoDB</strong>.<strong>Converter</strong>.unmarshall(record.<strong>dynamodb</strong>.NewImage)<br>    );<br>        <strong>for </strong>(let record of unmarshalledRecords) {<br>            <em>// this could be changed to any time window (a month, a year) and, in theory, the below algorithm would still work<br>            </em><strong>const </strong>MS_IN_DAY = 1000 * 60 * 60 * 24;<br>            <strong>const </strong>recordEndTime = <strong>new Date</strong>(record.TransferTime);<br>            <strong>const </strong>aggregatedDay = recordEndTime.getTime() - recordEndTime.getTime() % MS_IN_DAY;<br>            <strong>const </strong>params = {<br>                <strong>TableName</strong>: <strong>&#39;DailyAggregateTable&#39;</strong>,<br>                <strong>Key</strong>: {<br>                    <strong>CustomerId</strong>: record.CustomerId,<br>                    <strong>Day</strong>: <strong>new Date</strong>(aggregatedDay).toISOString(),<br>                },<br>                <strong>UpdateExpression</strong>: <strong>&#39;ADD #TotalBytes :Bytes, #CombinedRecords :CombinedRecords&#39;</strong>,<br>                <strong>ExpressionAttributeNames</strong>: {<br>                    <strong>&#39;#TotalBytes&#39;</strong>: <strong>&#39;TotalBytes&#39;</strong>,<br>                    <strong>&#39;#CombinedRecords&#39;</strong>: <strong>&#39;CombinedRecords&#39;</strong>,<br>                },<br>                <strong>ExpressionAttributeValues</strong>: {<br>                    <strong>&#39;:Bytes&#39;</strong>: record.Bytes,<br>                    <strong>&#39;:CombinedRecords&#39;</strong>: 1<br>                },<br>            };<br>            <strong>try </strong>{<br>                <em>//Write updates to daily rollup table<br>                </em>await documentClient.update(params).promise();<br>            } <strong>catch </strong>(err) {<br>                <em>//Swallow any errors<br>                </em><strong>console</strong>.error(<br>                    <strong>`Internal Error: Error updating dynamoDB record with keys [</strong>${<strong>JSON</strong>.stringify(<br>                        params.<strong>Key<br>                    </strong>)}<strong>] and Attributes [</strong>${<strong>JSON</strong>.stringify(params.<strong>ExpressionAttributeValues</strong>)}<strong>]`<br>                </strong>);<br>                <strong>console</strong>.error(err);<br>            }<br>        }<br>        callback(<strong>null</strong>, <strong>`Successfully processed </strong>${event.<strong>Records</strong>.<strong>length</strong>}<strong> records.`</strong>);<br>    } <strong>catch </strong>(err) {<br>        <strong>console</strong>.error(<strong>`Error processing records. Event was [</strong>${<strong>JSON</strong>.stringify(event)}<strong>`</strong>);<br>        <strong>console</strong>.error(err);<br><em>//Note we don&#39;t actually fail the lambda function here by calling back with the error e.g. callback(err)<br>        </em>callback(<strong>null</strong>, <strong>`Swallowed the error </strong>${<strong>JSON</strong>.stringify(err)}<strong>`</strong>);<br>    }<br>}<br>exports.handler = handler;</pre><p>A few snippets of interest</p><pre><strong>const </strong>insertRecords = event.<strong>Records</strong>.filter(record =&gt; record.eventName === <strong>&#39;INSERT&#39;</strong>);</pre><p>Here we are filtering the records down to just INSERT events. In theory you can just as easily handle DELETE events by removing data from your aggregated table or MODIFY events by calculating the difference between the old and new records and updating the table</p><pre><strong>UpdateExpression</strong>: <strong>&#39;ADD #TotalBytes :Bytes, #CombinedRecords :CombinedRecords&#39;</strong>,<br><strong>ExpressionAttributeNames</strong>: {<br>   <strong>&#39;#TotalBytes&#39;</strong>: <strong>&#39;TotalBytes&#39;</strong>,<br>   <strong>&#39;#CombinedRecords&#39;</strong>: <strong>&#39;CombinedRecords&#39;</strong>,<br>},<br><strong>ExpressionAttributeValues</strong>: {<br>   <strong>&#39;:Bytes&#39;</strong>: record.Bytes,<br>   <strong>&#39;:CombinedRecords&#39;</strong>: 1<br>},</pre><p>DynamoDB doesn’t support record-level locking, so how do you ensure that two lambda functions writing the same record at the same time they don’t both overwrite the initial value instead correctly aggregating both values? One answer is to use <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html">update expressions</a>. Here we are using an update expression to atomically add to the pre-existing <em>Bytes</em> value. Two, near-simultaneous, updates will successfully update the aggregated value without having to know the previous value.</p><p>There are a few different ways to use update expressions. For example, if you wanted to add a <em>createdOn</em> date that was written on the first update, but then not subsequently updated, you could add something like this to your expression:</p><pre><strong>SET #createdOn = if_not_exists(#createdOn, :now)</strong></pre><pre><strong>try </strong>{<br>   <em>//Write updates to daily rollup table<br>   </em><strong>await </strong>documentClient.update(params).promise();<br>} <strong>catch </strong>(err) {<br>   <em>//Swallow any errors<br>   </em><strong>console</strong>.error(err);<br>}</pre><pre>...</pre><pre><em>//Note we don&#39;t actually fail the lambda function here by calling back with the error e.g. callback(err)<br></em>callback(<strong>null</strong>, `Swallowed the error ${JSON.stringify(err)}`);</pre><p>Here we are swallowing any errors that occur in our function and not triggering the callback with an error. E.g. Not calling <em>callback(err)</em>. This is because your Lambda will get triggered with a batch of events in a single invocation (this can be changed by setting the <a href="https://docs.aws.amazon.com/lambda/latest/dg/API_CreateEventSourceMapping.html#API_CreateEventSourceMapping_RequestSyntax">BatchSize</a> property of the Lambda DynamoDB Stream event source), and you generally don’t want to fail the entire batch. This will be discussed more below.</p><h3>Gotchas and Lessons Learned</h3><h4>Kinesis, Batch Size, Error Handling, and Partial Failure</h4><p>Under the hood, DynamoDB uses <a href="https://aws.amazon.com/kinesis/data-streams/">Kinesis</a> to stream the database events to your consumer. By its nature, Kinesis just stores a log of events and doesn’t track how its consumers are reading those events. It simply provides an interface to fetch a number of events from a given point in time. It’s up to the consumer to track which events it has received and processed, and then request the next batch of events from where it left off (luckily AWS hides this complexity from you when you choose to connect the event stream to a Lambda function). This is a different paradigm than <a href="https://aws.amazon.com/sqs/">SQS</a>, for example, which ensures that <em>only one </em>consumer can process a given message, or set of messages, at a given time. In SQS you can then delete a single message from the queue so it does not get processed again. In Kinesis there is no concept of deleting an event from the log.</p><p>The inability to control the set of events that is coming from the stream introduces some challenges when dealing with errors in the Lambda function. <strong>There is no concept of a partial success</strong>. I.E. you can’t send information back to the stream saying: “I processed these 50 events successfully, and these 50 failed, so please retry the 50 that failed”. If you fail your entire Lambda function, the DynamoDB stream will resend the entire set of data again in the future. This is problematic if you have already written part of your data to the aggregate table. How do you prevent duplicate records from being written? And how do you handle incoming events that will never succeed, such as invalid data that causes your business logic to fail? There is no silver bullet solution for this case, but here are some ideas:</p><ul><li>Do some data-sanitization of the source events. If you can identify problems and throw them away before you process the event, then you can avoid failures down-the-line. Log the failures and possibly set up some CloudWatch Alarms to notify you of these unexpected cases. We used <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringPolicyExamples.html">CloudWatch Metric Filters</a> to translate our log errors into metrics we could observe and trigger alarms for.</li><li>Perform retries and backoffs when you encounter network or throughput exceptions writing to the aggregate table. This provides you more opportunity to succeed when you are approaching your throughput limits. If you are using an AWS SDK you get this <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.RetryAndBackoff">for free</a>.</li><li>If all else fails, write the event you are currently processing to some secondary storage. You cannot throw away this data if you want your destination table to be an accurate aggregate of the source table. Writing the event to an SQS queue, or S3, or even another table, allows you to have a second chance to process the event at later time, ideally after you have adjusted your throughput, or during a period of lighter usage. We implemented an SQS queue for this purpose.</li><li>Set your BatchSize to 1. I wouldn’t generally recommend this, as the ability to process and aggregate a number of events at once is a huge performance benefit, but it would work to ensure you aren’t losing data on failure. Simply trigger the Lambda callback with an error, and the failed event will be sent again on the next invocation. With this approach you have to ensure that you can handle events quickly enough that you don’t fall too far behind in processing the stream. A DynamoDB stream will only persist events for 24 hours and then you will start to lose data. You can monitor the <a href="https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html">IteratorAge</a> metrics of your Lambda function to determine how far behind you might be.</li></ul><h4>DynamoDB Throughput, Concurrency, Partitions, and Batch Writes</h4><p>Although DynamoDB is mostly hands-off operationally, one thing you <em>do </em>have to manage is your read and write throughput limits. Setting these to the correct values is an inexact science. Set them too low and you start getting throughput exceptions when trying to read or write to the table. Set them too high and you will be paying for throughput you aren’t using. <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html">Auto-scaling</a> can help, but won’t work well if you tend to read or write in bursts, and there’s still no guarantee you will never exceed your throughput limit.</p><p>In our scenario we specifically care about the write throughput on our aggregate table. We want to allow our Lambda function to successfully write to the aggregate rows without encountering a throughput exception. The logical answer would be to set the write throughput on the aggregate table to the same values as on the source table. After all, a single write to the source table should equate to a single update on the aggregate table, right? Unfortunately, the answer is a little more complicated than that.</p><p>First, you have to consider the number of Lambda functions which could be running in <em>parallel</em>. E.g. if you are running two Lambdas in parallel you will need double the throughput that you would need for running a single instance. The potential number of Lambdas that could be triggered in parallel for a given source table is actually based on the number of database <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.Partitions.html"><em>partitions</em></a><em> </em>for that table. There is one stream per partition. Unfortunately there is no concrete way of knowing the exact number of partitions into which your table will be split. It is a factor of the total provisioned throughput on the table and the amount of data stored in the table that roughly works out to something like</p><pre>Total partitions = <br>  MAX(Total partitions for desired performance, <br>      Total partitions for desired capacity)</pre><p>See this <a href="https://shinesolutions.com/2016/06/27/a-deep-dive-into-dynamodb-partitions/">article</a> for a deeper dive into DynamoDB partitions. You can get a rough idea of how many Lambda functions are running in parallel by looking at the number of separate CloudWatch logs your function is generating at any given time. There should be about one per partition assuming you are writing enough data to trigger the streams across all partitions.</p><p>You can also manually control the <a href="https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html">maximum concurrency</a> of your Lambda function. For example, if you tend to write a lot of data in bursts, you could set the maximum concurrency to a lower value to ensure a more predictable write throughput on your aggregate table. Again, you have to be careful that you aren’t falling too far behind in processing the stream, otherwise you will start to lose data.</p><p>Secondly, if you are writing to the source table in batches using the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html">batch write</a> functionality, you have to consider how this will affect the number of updates to your aggregate table. For example, a batch write call can write up to 25 records at a time to the source table, which could conceivably consume just 1 unit of write throughput. This will translate into 25 separate INSERT events on your stream. Since updating an item with update expressions cannot be done in batches, you will need to have 25x the throughput on the destination table to handle this case.</p><p>There is opportunity for optimization, such as combining the batch of events in memory in the Lambda function, where possible, before writing to the aggregate table. In practice, we found that having the write throughput on the aggregate table set to twice that of the source comfortably ensures we will not exceed our limits, but I would encourage you to monitor your usage patterns to find the number that works for your case.</p><h3>In conclusion</h3><p>Using the power of DynamoDB Streams and Lambda functions provides an easy to implement and scalable solution for generating real-time data aggregations. As a bonus, there is little to no operational overhead. The pattern can easily be adapted to perform aggregations on different bucket sizes (monthly or yearly aggregations), or with different properties, or with your own conditional logic. You could even configure a separate stream on the aggregated daily table and chain together multiple event streams that start from a single source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*oQ_h-g3FM--N0uj8.jpg" /></figure><p>There are a few things to be careful about when using Lambda to consume the event stream, especially when handling errors. Understanding the underlying technology behind DynamoDB and Kinesis will help you to make the right decisions and ensure you have a fault-tolerant system that provides you with accurate results.</p><p>Happy streaming!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f93547cfb244" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/real-time-aggregation-with-dynamodb-streams-f93547cfb244">Real-Time Aggregation with DynamoDB Streams</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A place for everything, everything in its place]]></title>
            <link>https://medium.com/signiant-engineering/a-place-for-everything-everything-in-its-place-bd045589908b?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/bd045589908b</guid>
            <category><![CDATA[lambda-function]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devsecops]]></category>
            <dc:creator><![CDATA[Dave North]]></dc:creator>
            <pubDate>Tue, 29 May 2018 17:11:04 GMT</pubDate>
            <atom:updated>2018-05-29T17:11:04.032Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*2ncTSQGqTxalVpJl.png" /></figure><p>Over the years, I’ve looked many times at <a href="https://www.datadoghq.com/">Datadog</a> and asked myself if it could be helpful to us at Signiant. Several times we’ve taken a look and, at the stage we were at, it just wasn’t a “must have”. Until it was. We found that as we broke down our big monoliths into smaller microservices, it was harder to easily visualize and track down errors. Further, while we had most of the data we needed, we had to mine it from multiple places (Cloudwatch, Papertrail, Jenkins, Bitbucket, Pingdom, Slack, Nagios, Docker, etc.). Datadog really gives us that great visualization, alerting and a “single pane of glass” where we can see what’s going on with various components.</p><p>After doing a thorough proof of concept, we purchased Datadog earlier this year and we’ve recently “finished” implementing Datadog across all our AWS infrastructure, (Elastic Beanstalks, ECS clusters, plain old instances). I say “finished” because there’s more we plan to pull in but we have enough there now to start with to get value. It’s already helped us solve some issues we were seeing just by having better visibility into our environments (the Docker instrumentation on ECS is fantastic)</p><h3>Secure Your DevOps (i.e. DevSecOps)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*HOF-c-UGs4FqZWQ0.jpg" /></figure><p>While we were implementing Datadog, we started using custom events to mark when we had promoted (released) code to production. This led us to ask if there were other events we could be capturing via use of custom events in Datadog.</p><p>We’re using<a href="https://www.trendmicro.com/en_ca/business/products/hybrid-cloud/deep-security-service.html"> Deep Security as a service</a> from <a href="/service/ttps://www.trendmicro.com/">Trend Micro</a> as our preferred cloud protection platform. It was straightforward to hook into our various AWS deployment models, captures a lot of IDS/IPS/log/filesystem events and pretty well works in the background to protect our environment and our SAAS applications. Deep Security works very well as an IDS/IPS system but it can generate rather a lot of events and is yet another console to login to. It turns out though that Deep Security can deliver an event stream to an AWS SNS topic and from there, we can do some processing on it. Like send it to Datadog. We came up with a solution that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*oDVmjn0iH4GRh_Sw0Ahl_A.png" /><figcaption>DS-&gt;SNS-&gt;Lambda-&gt;DD</figcaption></figure><p>Events come from Deep Security to the SNS topic where we have a Lambda function subscribed. This Lambda processes the event and sends it to Datadog while also allowing us to do some event filtering (ie. only send events with a rank greater than 50). In Datadog we then see something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AKEtIQhwEooITgdR." /></figure><p>Like any other events in Datadog, these can be alerted on, placed on dashboards, etc. It’s incredibly helpful to be able to overlay events on time boards within Datadog when looking into what may be happening at a time period.</p><p>The solution is all packaged up in a nice Cloudformation template and available in a Github repo <a href="https://github.com/Signiant/datadog-deepsecurity-event-forwarder">Signiant/datadog-deepsecurity-event-forwarder</a></p><p>How are you doing DevSecOps with smaller teams? What’s working (or not working!) for you?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd045589908b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/a-place-for-everything-everything-in-its-place-bd045589908b">A place for everything, everything in its place</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compliance at Velocity with AWS Config]]></title>
            <link>https://medium.com/signiant-engineering/compliance-at-velocity-with-aws-config-8edcc3a3d04d?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/8edcc3a3d04d</guid>
            <category><![CDATA[amazon-ec2]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[configuration-management]]></category>
            <category><![CDATA[config]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Mike Perttula]]></dc:creator>
            <pubDate>Wed, 14 Feb 2018 21:14:36 GMT</pubDate>
            <atom:updated>2018-02-20T14:11:14.475Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*XmL6XoaeClJ6wfwrAB2TSA.jpeg" /><figcaption>..in most cases</figcaption></figure><p>When it’s time to lay down the law, you better make sure you close the loopholes. This is what we had to do recently when we noticed that our <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html">EC2 Security Groups</a> were skirting compliance in our development environments. The rule in this case is don’t allow any open internet connections (0.0.0.0/0) on SSH port 22 or RDP port 3389. Any service or developer wanting an instance to be accessible via SSH or RDP had to make sure that the connection was locked down to a known <a href="https://www.signiant.com/">Signiant</a> network. We found that there were outlaws skirting these rules and thats where we sprang into action.</p><h4>Creating Compliance</h4><p>At re:Invent in 2014, AWS introduced the <a href="https://aws.amazon.com/config/">Config service</a>. The service works as watchdog, auditing changes in your AWS account and alerting on whether your service configurations fall into a compliance with the standards of the environment. This is determined by rules that you create to track down what you consider compliant. A couple of years ago we decided to make use of this service to lock down EC2 Security to groups in an automated fashion. As mentioned earlier, we wanted to make sure that anyone creating Security Groups would need to make sure that SSH and RDP was locked down. To do this we created 2 config rules.</p><p>First we would tackle SSH. The setup of a rule is trivial with AWS Config’s libary of Managed Rules. When Clicking the “Rules” link in the Config dashboard and choosing “Add rule”, we are given the option of viewing managed rules. These rules are provided by AWS as common configuration audits us to use. There is already a restricted-ssh option for us to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*x6MYzU5W7f_Ns8G7uwPuxA.png" /><figcaption>A library of rules at your fingertips</figcaption></figure><p>Choosing this rule gives us everything we wanted and will alert on non-compliance should a Security Group be created with a wide open SSH inbound rule. The default behaviour for the rule’s frequency is to check for compliance every time a configuration change occurs on a Security Group within the account. From here we need a compliance officer to action non compliant groups and making everything on the level.</p><h4>Reaping Rewards</h4><p>The offenders have been identified by AWS Config but we still need to action that intel. AWS documentation suggests <a href="https://docs.aws.amazon.com/config/latest/developerguide/notifications-for-AWS-Config.html">streaming config changes to SNS</a> and then subscribing to that topic via email or any other alerting method of your choice. This is a good method to be informed of the issue but we needed to go further and handle the alert in automated fashion, shutting down the rule. For that we created a solution called the <a href="https://github.com/Signiant/aws-config-rule-port-reaper">AWS Config Rule Port Reaper</a>, a docker container that is ready to do the dirty work of shutting down rogue open ports. It is scheduled to run periodically in each of our AWS accounts, finding out what resource isn’t compliant and then removing the rule in question based on the type of infraction.</p><p>We had originally developed this before Lambda was being widely used at Signiant. Today this would most likely be a Lambda function from the get go, weeding out trouble and returning us to a fairly secure state.</p><h4>Catching Them All</h4><p>Until recently everything was working well with our security group Compliance. Users were more mindful on what ports they left opened and we rarely had the reaper doing working. But then we found a loophole.</p><p>We received an alert that the reaper ran and that there was a Security Group not in compliance. We expected to the Reaper to work as per normal but something was amiss. We continued to receive warnings about the same security group being not in compliance for SSH. This should have been handled by our ‘Lawbringer’ but it was not making the cut. Taking a closer look at the security group we found not every case was being considered. The group looked as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*uG6hyo9M1RlAJZt8mex6pQ.png" /><figcaption>The Loophole</figcaption></figure><p>SSH was wide open, and the Config rule had deemed it not compliant. It seems our Reaper tool was not robust enough to handle a port range over a port that was being monitored. Further to this, we had used the managed AWS config rule, <a href="https://console.aws.amazon.com/config/home?region=us-east-1#/rules/rule-details/restricted-rdp">restricted-rdp, </a>to alert on RDP ports that open to the web. We learned this rule did not take into account that RDP can be over TCP and UDP, alerting only on TCP cases. The TCP security group rule was squashed by the reaper but the UDP rule was left behind in a state of non conformity. The oversight in the Reaper would require a quick change in checking for a range condition, but the restricted-rdp over UDP was going to require a Custom Rule.</p><p>There are a few ways to create a custom Config rule. You can go the manual route where you follow the AWS Config console creation wizard which includes you developing a Lambda from scratch ( or using an existing one ). It has to handle the Config event that gets fired by the rule for checks. The event includes a “Configuration Item” object that will includes all the configuration of a single AWS Resource in json. In our case it’s a Security Group configuration. As well you will be able to pull any parameters specified in the Config rule parameters section that can optionally be used with the evaluating code. The Lambda must return either COMPLIANT, NON_COMPLIANT, or NOT_APPLICABLE (for those times you are way off in your evaluation). Figuring out a manual deploy and linking the ARN back to the rule is cumbersome and we also have to think about how we are going to roll this out to multiple accounts and multiple regions. Luckily I was able to come across a speedy solution to my woes.</p><h4>Ruling at Velocity</h4><p>The stranger that came into town to clean up the mess ended up being the <a href="https://github.com/awslabs/aws-config-rdk">AWS Config Rules Development Kit</a> provided by <a href="https://github.com/awslabs">AWS Labs</a>. The python based tool, available on to install on pip, allows you to ramp up rule creation in a flash. Once installed, you initialize your working directory where you’ll be writing the rule which also prepares an S3 bucket for the region you’ll be deploying the rule:</p><pre>rdk -p dev -r us-east-1 init</pre><p>Your lambda code will also be stored in this bucket. Next you create a rule:</p><pre>rdk -p dev create restrict-rdp-udp --runtime python2.7 --resource-types AWS::EC2::SecurityGroup --input-parameters &#39;{&quot;PORT&quot;:&quot;3389&quot;}&#39;</pre><p>This automatically generates stub code in my language of choice (java, nodejs, or python), in this case python, and will be checking the SecurityGroup resource type. We also introduce the PORT input parameter. All that’s left to do is to look through the Configuration object and parameters to determine what I think would be complaint. The stub has a section in the evaluate_complaince method for you to determine what is complaint vs non complaint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*IT9KvqLm2Q2emsN6piOhxQ.png" /><figcaption>Stub code generated by the RDK</figcaption></figure><p>The code that I ended up using is detailed in my <a href="https://bitbucket.org/signiant/custom_config_rules">Custom Config Rule repository</a>. You can test your code with an example resouce, test_event_template.json, that they provide. Once you’re satisfied, it’s time to deploy:</p><pre>rdk -p dev -r us-east-1 deploy restrict-rdp-udp</pre><p>This command uploads a compressed Lambda to the initialized Config bucket, and runs a CloudFormation Template to create the rule and deploy the Lambda. The use of this suite help me with the majority of the heavy lifting and the deploy was very fast and repeatable to multiple regions and/or accounts. I had started with the manual processes but easily abandoned it once I found this tool.</p><p>The AWS Config RDK is considered to be in “Open Beta” right now so you may want to evaluate it for your purposes. What’s locked down in the Lambda role creation may be something you want to review for a production environment. Otherwise I recommend it for the speed of compliance it allowed me for something that was missing.</p><p>A future tool I plan to evaluate is <a href="https://github.com/iagcl/watchmen">Watchmen</a>, which allows you to centrally manage Config Rules in one account instead of having to deploy them to multiple regions and accounts. This looks like an ideal solution to speed up where your rules are deployed saving you some time on making sure your environments are on lockdown.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8edcc3a3d04d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/compliance-at-velocity-with-aws-config-8edcc3a3d04d">Compliance at Velocity with AWS Config</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Free Cleaning with AWS DynamoDB TTLs]]></title>
            <link>https://medium.com/signiant-engineering/free-cleaning-with-aws-dynamodb-ttls-c396facb9937?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/c396facb9937</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dave North]]></dc:creator>
            <pubDate>Wed, 25 Oct 2017 17:57:13 GMT</pubDate>
            <atom:updated>2017-10-25T20:18:05.717Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*LY1aYgw66mj4iZIK-318jw.png" /><figcaption>Clean all the things (and fish)</figcaption></figure><p>I’ve written before about how easy DynamoDB is to manage and store data. Lots of data. Sometimes, too much data! Given how inexpensive storage is for DynamoDB, it’s tempting to just leave data in there forever. However, this eventually catches up with you if you ever need to do table scans in the form of an ever more increasing amount of provisioned throughput required.</p><p>It’s possible to have a series of scripts, code, etc to prune data from tables with simple deletes matching some criteria. But these consume write throughput and it feels very “old school”. What if there was some way to prune data from tables “for free”? There is! Back in Feb 2017, AWS added a feature called “TTLs” to DynamoDB which we’ve just started using and in the words of <a href="http://seinfeld.wikia.com/wiki/Kenny_Bania">Kenny Bania</a>, it’s “Gold Jerry, Gold”</p><h3>Set it and…forget it!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*NBl6WKaf2OucphjQsfeC4g.png" /></figure><p>I hate to use a tagline from a <a href="https://medium.com/signiant-engineering/set-it-and-forget-it-a66914eacc02">previous blog article</a> but, like DynamoDB autoscaling, it’s another case of “turn it on, go to sleep, magic happens”. TTLs are really easy to start using as long as you have an attribute in the table that can be used as an expiry time. The basic workflow with TTLs is:</p><ol><li>Designate an attribute in the table that contains the expiry time. This must be an epoch time</li><li>There is no step 2. DynamoDB prunes data in the background <strong><em>WITHOUT</em></strong> using any provisioned throughput</li></ol><h3>Our use case</h3><p>One of our main products/services is <a href="https://www.signiant.com/products/media-shuttle/">Media Shuttle</a> which is a high speed file transfer solution for very large files. When a user sends a file or collection of files to another user, the recipient receives an email informing them they can download. We send these emails today with AWS Simple Email Service (SES) which is..uhh…simple. SES allows you to capture records of all deliveries and bounces to a Simple Notification Service (SNS) topic where you can email them or process them.</p><p>We’ve <a href="https://aws.amazon.com/blogs/apn/how-signiant-uses-aws-lambda-and-amazon-dynamodb-to-run-its-saas-solution-on-aws/">written with AWS</a> about how we process these delivery and bounce records with a Lambda function and write them to DynamoDB in the past. These meta-records are used almost exclusively by our product support specialists to help in troubleshooting calls of the form “I did not get my email”. Usually these calls are “near term” to when a delivery has been sent so the lifespan of having these email delivery or bounce records is pretty short. This solution looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*jMMoeJzRxADFQmO6Z2ahRQ.png" /><figcaption>Processing SES bounces and deliveries with Lambda</figcaption></figure><p>We figured we’d just keep the records forever in DynamoDB since they are small and not queried every day. However, the deliveries table has now grown pretty large and as we allow wildcard queries on this table with our GUI tool, it means a table scan which consumes quite a large amount of provisioned throughput. What if we could trim this data automatically…if only there were TTLs.</p><h3>Hammertime</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*CKLwSZvYfbLXt4EeSAjjAQ.png" /></figure><p>To enable TTLs for this existing application, we had to do 3 things:</p><ul><li>Modify the <a href="https://github.com/Signiant/aws-ses-recorder">existing Lambda functions</a> to add a new attribute to the tables whenever we inserted a new record. This was as simple as adding a quick calculation to insert a value 120 days in the future:</li></ul><pre>future = datetime.datetime.utcnow() + datetime.timedelta(days=120)<br>expiry_ttl = calendar.timegm(future.timetuple())<br>:<br>:</pre><pre># Add entry to DB for this recipient<br>Item={<br>    &#39;recipientAddress&#39;: recipientEmailAddress,<br>    &#39;sesMessageId&#39;: sesMessageId,<br>    &#39;sesTimestamp&#39;: long(sesTimestamp_seconds),<br>    &#39;deliveryTimestamp&#39;: long(deliveryTimestamp_seconds),<br>    &#39;processingTime&#39;: long(processingTime),<br>    &#39;reportingMTA&#39;: reportingMTA,<br>    &#39;smtpResponse&#39;: smtpResponse,<br>    &#39;sender&#39;: sender.lower(),<br>    &#39;expiry&#39;: long(expiry_ttl)<br>}</pre><ul><li>Enable TTLs on the table and tell it to use the new expiry attribute as the TTL</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*Gj61mBfI5P0HEWpFO9ChSg.png" /><figcaption>Enabling TTL for table</figcaption></figure><ul><li>Finally, we wanted to backfill all existing records so they would be expired. Luckily, we already had an attribute in the database that contains the timestamp the record was added. So we created a small script to update all records that did not have an expiry with one 120 days in the future. It’s not the prettiest but basically all we do is scan the table using the filter expression attribute_not_exists(expiry). Then for each record returned, we just pop in an expiry. Alas, there is no batch update functionality so this has to process each record sequentially. Here’s what we used:</li></ul><pre>import boto3<br>import datetime<br>import calendar</pre><pre>def get_expiry(added_time):<br>    future = datetime.datetime.fromtimestamp(float(added_time)) + datetime.timedelta(days=120)<br>    expiry_ttl = calendar.timegm(future.timetuple())</pre><pre>return expiry_ttl</pre><pre>def update_item(item, expiry, client, table_name):<br>    recipient = item[&#39;recipientAddress&#39;][&#39;S&#39;]<br>    message_id = item[&#39;sesMessageId&#39;][&#39;S&#39;]<br>    print &quot;updating item &quot; + recipient + &quot; / &quot; + message_id + &quot; with expiry &quot; + str(expiry)</pre><pre>response = client.update_item(<br>        TableName=table_name,<br>        Key={<br>            &#39;recipientAddress&#39;: {<br>                &#39;S&#39;: recipient,<br>            },<br>            &#39;sesMessageId&#39;: {<br>                &#39;S&#39;: message_id,<br>            },<br>        },<br>        ExpressionAttributeNames={<br>            &#39;#ED&#39;: &#39;expiry&#39;<br>        },<br>        ExpressionAttributeValues={<br>            &#39;:ed&#39;: {<br>                &#39;N&#39;: str(expiry),<br>            }<br>        },<br>        ReturnValues=&#39;ALL_NEW&#39;,<br>        UpdateExpression=&#39;SET #ED = :ed&#39;<br>    )</pre><pre>if &#39;ResponseMetadata&#39; in response:<br>        if &#39;HTTPStatusCode&#39; in response[&#39;ResponseMetadata&#39;]:<br>            if response[&#39;ResponseMetadata&#39;][&#39;HTTPStatusCode&#39;] != 200:<br>                print &quot;** Error updating &quot; + recipient + &quot; / &quot; + message_id<br>            else:<br>                print &quot;+ Successfully updated &quot; + recipient + &quot; / &quot; + message_id<br>        else:<br>            print &quot;** No http status code in response - error &quot;<br>    else:<br>        print &quot;** No response metadata&quot;</pre><pre>if __name__ == &quot;__main__&quot;:<br>    table_name = &#39;SES_BOUNCES&#39;<br>    region = &#39;us-east-1&#39;</pre><pre>session = boto3.session.Session(region_name=region)<br>    dynamodb_client = session.client(&#39;dynamodb&#39;)</pre><pre># This will give us all entries without an expiry set<br>    filter_expression = &quot;attribute_not_exists(expiry)&quot;</pre><pre>response = dynamodb_client.scan(<br>        TableName=table_name,<br>        FilterExpression=filter_expression,<br>    )</pre><pre>for item in response[&#39;Items&#39;]:<br>        create_time = item[&#39;sesTimestamp&#39;][&#39;N&#39;]<br>        expiry_ttl = get_expiry(create_time)<br>        print &quot;Item was added on &quot; + str(create_time) + &quot; and expires on &quot; + str(expiry_ttl)<br>        update_item(item, expiry_ttl, dynamodb_client, table_name)</pre><pre>while &#39;LastEvaluatedKey&#39; in response:<br>        response = dynamodb_client.scan(<br>            TableName=table_name,<br>            FilterExpression=filter_expression,<br>            ExclusiveStartKey=response[&#39;LastEvaluatedKey&#39;]<br>        )</pre><pre>for item in response[&#39;Items&#39;]:<br>            create_time = item[&#39;sesTimestamp&#39;][&#39;N&#39;]<br>            expiry_ttl = get_expiry(create_time)<br>            print &quot;Item was added on &quot; + str(create_time) + &quot; and expires on &quot; + str(expiry_ttl)<br>            update_item(item, expiry_ttl, dynamodb_client, table_name)</pre><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*bxI3e0sVwAFpsK5ThSPQAA.png" /></figure><p>The backfill took a LONG time to run as there are millions of rows in the table. However, once it was complete I came into the office the next day to see 1 million rows had been “expired”. Free. No provisioned throughput used. No Spartans were harmed in this expiry.</p><p>One other case we looked into is saving off the data that was removed by the TTL process. It turns out that you can <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-streams.html">identify records removed by a TTL</a> in the dynamo streams so we could have a trigger (Lambda) that looks for these and writes them to S3, Redshift, etc. That’s not a requirement for this specific use case but it’s nice to know the option is there as we have other cases where this will almost certainly be a requirement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c396facb9937" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/free-cleaning-with-aws-dynamodb-ttls-c396facb9937">Free Cleaning with AWS DynamoDB TTLs</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set it and…Forget it]]></title>
            <link>https://medium.com/signiant-engineering/set-it-and-forget-it-a66914eacc02?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/a66914eacc02</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[autoscaling]]></category>
            <dc:creator><![CDATA[Dave North]]></dc:creator>
            <pubDate>Thu, 21 Sep 2017 14:24:03 GMT</pubDate>
            <atom:updated>2017-09-21T14:25:00.685Z</atom:updated>
            <content:encoded><![CDATA[<p>DynamoDB autoscaling in action</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*6VKH9cQeTIO9AYYQU9gYkw.png" /></figure><p>We’ve been using AWS DynamoDB for a while at Signiant as our main data store for our SAAS applications. In fact, we were actually on the original beta program for it many years ago now. DynamoDB does come with a bunch of challenges in using it but there’s no doubt in my mind, it reduces the ops load of running a database by a factor of 90%. It just works, requires none of the traditional database maintenance a relational database requires (postgres vacuum anyone?!) and performance is great. There is one problem though….</p><h3>Spikes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*4cumnR1wdZRYDcn0ANghEg.png" /></figure><p>Spikes. Read and write spikes. DynamoDB tables are setup not by how much storage capacity you need but by how much read and write capacity you need. This is expressed in terms of provisioned read and write throughput. You can burst over your provisioned limit for a short period but in general, if you hit the provisioned rate, you’ll be told to go away and slow down your read or write rate. Take a look at this table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*A7ea3YaWt9-lWarXiSr0Hw.png" /></figure><p>Here, the write pattern to the table is very uneven and “spiky”. This can cause havoc if your application cannot gracefully handle this. With a single table, it’s manageable to manually increase provisioned throughput but in our case, we have close to 150 DynamoDB tables so that clearly wasn’t going to work (unless we had a room of monkeys with keyboards!)</p><h3>Autoscale All the Things</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*rflR-rozSH_r2YK4bLIBXA.png" /></figure><p>Back in 2013, we solved this problem by writing our own DynamoDB autoscaler. It worked by waking up every 5 minutes, getting a list of all tables, checking the current read or write throughput and automatically increasing the provisioned throughput if the current use was high enough. This worked fantastically well and solved a good number of issues we were continually having in various parts of our applications (ref: the number of tables we have) that were caused by throughput exceptions. Earlier this year though, AWS introduced autoscaling functionality into DynamoDB using the application autoscaling functionality. After doing some testing with this, it was clearly superior to our old (but well used!) tool in that it was configured per table and it could scale throughput down. We did not build scale down into our original tool because while you can scale up a table an unlimited amount per day, you can only scale down a limited number of times per day. In 2013, that was 2 times. It’s currently 8 times per day.</p><h3>Migrating to Native DynamoDB Autoscaling</h3><p>As with most things, moving to a new way of doing things wasn’t quite a matter of checking a box. While the AWS console is literally a checkbox on each table to enable auto-scaling, we have 150 tables. And our dev teams regularly add new tables so we want autoscaling automatically enabled for new tables. Oh and we’d like to know when tables are scaling.</p><p>It turns out that while it’s a single check-box in the AWS console, like a lot of AWS features, there’s quite a bit going on behind the curtain. DynamoDB autoscaling relies on a feature called application autoscaling (this is also used for ECS autoscaling) while requires configuring a few things to make this work. Enabling this on 150+ tables was going to be some work, so we created a <a href="https://github.com/Signiant/dynamodb-autoscale-enabler">small utility</a> to enable/disable autoscaling on groups of tables. This was a great learning exercise as it allowed us to see exactly how autoscaling worked behind the curtain.</p><p>Here’s what a table looks like now we have autoscale enabled for read:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*LBSChuldvLZ7XiCFH938Sg.png" /></figure><h3>Always Autoscale</h3><p>After enabling autoscaling on the existing tables and making sure it works as expected, we wanted to make sure that all new tables have autoscale enabled on them. We currently create tables using cloudformation as part of our deployment pipeline so the first thought was to have dev teams add the autoscaling option to the templates. However, the amount of extra bits and pieces to add per table (targets, scaling policies) is pretty ornery (especially if your table has indexes). So we crated a <a href="https://github.com/Signiant/dynamodb-autoscale-lambda-enable">lambda function hooked up to a Cloudwatch event</a>. This is triggered whenever a new table is added and checks if autoscaling is enabled for the table. If not, it turns it on.</p><p>The interesting learning experience here was how granular you can get with Cloudwatch API events. Take a look at the event pattern:</p><pre>{<br>  &quot;source&quot;: [<br>    &quot;aws.dynamodb&quot;<br>  ],<br>  &quot;detail&quot;: {<br>    &quot;eventSource&quot;: [<br>      &quot;dynamodb.amazonaws.com&quot;<br>    ],<br>    &quot;eventName&quot;: [<br>      &quot;CreateTable&quot;<br>    ]<br>  }<br>}</pre><p>Here, we’re able to have the event fire a target if it’s coming from dynamodb, only for the CreateTable API action. When this condition is met, the Lambda function is executed which does the leg work to ensure that autoscaling is enabled on “this” table and/or indexes.</p><h3>See the Scaling</h3><p>Our legacy table autoscaling tool used to send us an email when it was manipulating throughput. This was handy really just as a “is this really working?” notification. Native DynamoDB autoscaling documentation talked about there being an SNS topic we could subscribe to but after talking to support, it seems this doesn’t actually exist. And who wants lots of email these days?</p><p>We use Slack. A lot. So what if we could get those notifications into a slack channel? Turns out, we can use a similar strategy to capturing the new table events using Cloudwatch events. In this case, we want to know when autoscaling has triggered an update to the table. AWS support gave us a pointer to what the event rule would look like which ended up being:</p><pre>{<br>  &quot;source&quot;: [<br>    &quot;aws.dynamodb&quot;<br>  ],<br>  &quot;detail&quot;: {<br>    &quot;eventName&quot;: [<br>      &quot;UpdateTable&quot;<br>    ],<br>    &quot;userAgent&quot;: [<br>      &quot;application-autoscaling.amazonaws.com&quot;<br>    ]<br>  }<br>}</pre><p>Here, we’re able to have the event fire a target if it’s coming from dynamodb, only for the UpdateTable API action and only if the service triggering the event is application-autoscaling. When this condition is met, a <a href="https://github.com/Signiant/dynamodb-autoscaling-to-slack">Lambda function</a> is executed which formats up a message to post to slack. The Slack output looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOGviuKkHhaDepn1Lp3P6Q.png" /></figure><p>We’ve found that with a large number of tables, DynamoDB autoscaling scales a lot. So we have a dedicated Slack channel for this bot to post to so our main channels are not full of autoscaling goodness</p><h3>There is…Just One More Thing</h3><p>We’ve <a href="https://medium.com/signiant-engineering/dynamodb-backup-with-emr-30ab35c2d559">written before</a> how we backup larger number of DynamoDB tables using EMR. With this process, we actually “spike” the provisioned read throughput of a table before we dump the content so we can backup the table in a reasonable time. Enabling autoscaling on this table just caused the throughput to be lowered right away. So we needed a way to pause autoscaling before we backup the table. This would be a great feature in DynamoDB but it’s not there today. What we ended up having to do was modify the scaling policy for each table before we back it up to set the minimum throughput to be our spiked value. You can see this change with <a href="https://github.com/Signiant/dynamodb-emr-exporter/commit/d790f5f702b535b75302483dc40d5dc3d372319c">this commit</a>. It would be great if there was some simple way to use pause autoscaling on a table though.</p><h3>Referenced Projects</h3><ul><li><a href="https://github.com/Signiant/dynamodb-autoscaling-to-slack">Signiant/dynamodb-autoscaling-to-slack</a></li><li><a href="https://github.com/Signiant/dynamodb-autoscale-enabler">Signiant/dynamodb-autoscale-enabler</a></li><li><a href="https://github.com/Signiant/dynamodb-autoscale-lambda-enable">Signiant/dynamodb-autoscale-lambda-enable</a></li><li><a href="https://github.com/Signiant/dynamodb-emr-exporter">Signiant/dynamodb-emr-exporter</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a66914eacc02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/set-it-and-forget-it-a66914eacc02">Set it and…Forget it</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DynamoDB Backup with EMR]]></title>
            <link>https://medium.com/signiant-engineering/dynamodb-backup-with-emr-30ab35c2d559?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/30ab35c2d559</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[emr]]></category>
            <dc:creator><![CDATA[Dave North]]></dc:creator>
            <pubDate>Tue, 18 Jul 2017 14:49:41 GMT</pubDate>
            <atom:updated>2017-07-18T14:51:50.830Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*P2CJ5TUmunabPH0jzvjZbw.png" /></figure><p>Backups. Everyone needs them, everyone hopes they never need them! We use AWS DynamoDB quite a bit at Signiant with close to 200 tables in production. Dumping a single table is pretty straightforward using AWS’ built-in solution that relies on data pipeline. We’ve gone through several iterations of how we backup this data in a reasonable time and have a solution that we’ve been using for a while now that seems to work</p><h3>In the Beginning</h3><p>When we first started using DynamoDB, we didn’t have a lot of tables or a lot of data. So we had a pretty simple solution of a .NET tool we had created that dumped table data to XML. This was just scheduled in the Windows scheduler as a batch file. It worked but we didn’t handle failures well and because all the tables were backed up sequentially, it started to get slow.</p><h3>More Reliability</h3><p>Our first cut at improving things was to wrap the .NET tool in a script which handled failures, retried operations, etc. This drastically improved our backup success rate but it was still getting slower as we got more data and added more tables.</p><h3>More Speed</h3><p>Next, we added some parallelism to the table dumps using some threads. This decreased the backup time by a huge factor but it was cumbersome to manage new tables being added and it all ran on a single EC2 instance writing to local storage which quickly became the bottleneck.</p><h3>The Rewrite</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*LaiSRa78FcFQjbdPnjr7mQ.png" /></figure><p>In true “<em>this is all crap, I need to re-write everything</em>” fashion so often found in the software development world, it was time for a re-think of the whole process. We took a look at using the DynamoDB/DataPipeline solution to dump the table data but that used a single Elastic Map Reduce (EMR) cluster per table. With over 200 tables to dump, that was going to get expensive. However, we were able to reverse-engineer what Data Pipeline was doing to dump the table and that led to the current solution. Which is…</p><ol><li>Using a small python/boto script, we grab a list of all the tables to be dumped and generate an EMR ‘step’ for dumping this table. The list of tables can be filtered by a prefix (ie. all tables beginning with PROD). Each EMR step uses the AWS provided library to access DynamoDB and write the dumped data to S3.</li><li>Within each step, we also have the option to “spike” the read throughput before the dump takes place and lower it when it’s complete. This saves a huge amount of time while at the same time, keeps the DynamoDB costs as low as possible for provisioned throughput</li><li>We then create a new EMR cluster and pass in the steps and the other EMR configuration data needed. The EMR cluster can be configured with more or less workers as needed and essentially does all the work at this stage. Clusters that fail to start are retried.</li><li>A watchdog monitors the state of the cluster and all it’s steps and reports the status at the end.</li></ol><p>In our case, we’re able to sub-divide our tables into 6 different “prefixes” so we can backup 6 sets of tables in parallel using 6 EMR clusters. All of the clusters are spun up on demand (using spot instances) and torn down when the dumps complete. Restores are just the reverse process.</p><p>We’ve been using this process for around 2 years now and it’s been incredibly reliable. Since switching to spot instances late in 2016, we’ve reduced the costs by around 90% while still keeping the same level of performance.</p><p>The scripts and ‘moving parts’ for this solution are available in a github repo: <a href="https://github.com/Signiant/dynamodb-emr-exporter">signiant/dynamodb-emr-exporter</a></p><h3>Bonus Content</h3><p>As well as making sure we have a backup of all our DynamoDB content, we also replicate it using a replication solution we developed. You can read all about that in <a href="https://medium.com/signiant-engineering/all-of-your-replication-servers-are-belong-to-us-765c9ec4d20d">All of your replication servers are belong to us</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30ab35c2d559" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/dynamodb-backup-with-emr-30ab35c2d559">DynamoDB Backup with EMR</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Highly Available Downloads Across Clouds]]></title>
            <link>https://medium.com/signiant-engineering/highly-available-downloads-across-clouds-23a9ff6df03d?source=rss----cc5a48dd2b94---4</link>
            <guid isPermaLink="false">https://medium.com/p/23a9ff6df03d</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[replication]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Mike Perttula]]></dc:creator>
            <pubDate>Tue, 04 Jul 2017 18:24:23 GMT</pubDate>
            <atom:updated>2017-07-04T18:24:23.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*yzIbm1pw19IMuhJVXA8txQ.jpeg" /></figure><p>We’re always told never to put all your eggs in one basket. If you have used Amazon’s Simple Storage Service (S3) in the past year then you may know why. On the morning of February 28th, 2017 Amazon had a large outage in their S3 subsystem that handles all their GET, LIST, PUT, and DELETE requests in the US-East-1 region. If you used Object storage in that region you were out of luck until the outage was resolved which, in this case, took hours. At Signiant, we have a redundancy strategy for our user-facing web applications but during the S3 outage, a flaw was highlighted in how we distribute our installable components. We went into into action to address this.</p><p>In doing our post mortem we determined our ability to distribute software and binaries to customers can be affected by an S3 outage. S3 has its hooks in much of the functionality of AWS and specifically Cloudfront, Amazon’s CDN solution, was affected by the outage. We use Cloudfront to serve up a number of our software assets such as the <a href="https://shuttle.support.signiant.com/customer/portal/articles/2469753-download-signiant-app">Signiant App</a> (which allows customers to make use of our acceleration technology in <a href="https://www.signiant.com/products/media-shuttle/?cc=homepage">Media Shuttle</a>). Not having new versions available, or being possibly offline if nothing could be served by Cloudfront, is not a risk we wanted to take.</p><h3>X - CDN</h3><p>The simplest way to make sure you are highly available for object storage is to make sure assets are served from more than one region. This is a staple at Signiant and we already have these assets served in multiple regions. With the S3 outage though, we found that all of Cloudfront was unavailable for a period of time so we decided to go cross CDN to ensure our object storage for downloadable assets is not dependant on a single CDN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*A_N0cEv6l_xXfDaH9eNv9w.png" /></figure><p>Signiant has a great relationship with AWS but also with Azure. Our <a href="https://www.signiant.com/products/media-shuttle/?cc=homepage">Media Shuttle</a> and <a href="https://www.signiant.com/products/flight/?cc=homepage">Flight</a> products both allow uploading and downloading from both AWS and Azure cloud storage. It was a no brainer in using Azure for our cross cloud redundancy given we already run infrastructure there today. Azure’s CDN service makes use of Akamai and Verizon for serving content so we get a bonus of possibly using up to 3 CDNs at one time if needed.</p><h3>Object storage Copy</h3><p>When releasing our downloadable software, we make use of our <a href="https://medium.com/signiant-engineering/a-clone-army-of-the-jenkins-republic-4b6568caf33">Jenkins </a>promotion tools to get the downloadable artifacts to S3. From there Cloudfront will serve them to its edge nodes for end user download. Instead of investing time in modifying the promotion tools to also upload to Azure blob storage, we created a small service called the “CDN Replication Service”. This service polls an Amazon SQS Queue for changes to an s3 bucket and uploads them to Azure blob storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*CugtJsKiPo3QA2GF6DVWHw.png" /><figcaption>S3 Events to SQS</figcaption></figure><p>S3 has a handy ability to send S3 events such has PUT/GET/COPY/DELETE to multiple Amazon Services. By forwarding the upload events to a queue, our replication service is free to pull from that queue and replicate files as they are added. Since our downloadable artifacts are public, we can send a command to Azure’s <a href="https://docs.microsoft.com/en-us/rest/api/storageservices/copy-blob">Copy Blob</a> REST call to pull the new or changed files directly from S3 to a storage location on Azure without incurring any middleman transfer charges from the service. Any transfer failures are left on the queue to retry on the next poll.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*J-eYh-zCY-tON9teQisQIg.png" /></figure><p>Currently the service only handles new or changed content. A future enhancement will be to handle messages related to deletion (not required at this time as we don’t prune old versions often).</p><h3>How available do you need to be?</h3><p>Our failover strategy for downloadable assets may not be for everyone. Here are some things to consider when determining your failover strategy:</p><ul><li>Cost : Is hosting across multiple CDNs and cloud storage worth the return you will receive in availability for customers?</li><li>Complexity : Are you adding additional layers of complexity and another point of failure with your HA solution?</li></ul><p>Any cloud solution is bound to have downtime. In Signiant’s case the cost and complexity was minimal and we felt adding this change does provide enough benefit so that our software is available for download 24/7.</p><p>If you are interested in our solution, you can find s3queue2blob code on <a href="https://github.com/Signiant/s3queue2blob">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23a9ff6df03d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/highly-available-downloads-across-clouds-23a9ff6df03d">Highly Available Downloads Across Clouds</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>