<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Peter Hodgkinson on Medium]]></title>
        <description><![CDATA[Stories by Peter Hodgkinson on Medium]]></description>
        <link>https://medium.com/@pdhodgkinson?source=rss-98789c49782e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fc6E8NP5QhU7BHmr7HYALw.jpeg</url>
            <title>Stories by Peter Hodgkinson on Medium</title>
            <link>https://medium.com/@pdhodgkinson?source=rss-98789c49782e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:17:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pdhodgkinson/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using DynamoDB transactions to model relational data]]></title>
            <link>https://medium.com/signiant-engineering/using-dynamodb-transactions-to-model-relational-data-c13619ca58ae?source=rss-98789c49782e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c13619ca58ae</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <dc:creator><![CDATA[Peter Hodgkinson]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 16:13:36 GMT</pubDate>
            <atom:updated>2019-09-06T13:37:12.214Z</atom:updated>
            <content:encoded><![CDATA[<p>At Signiant we use AWS’s <a href="https://aws.amazon.com/dynamodb">DynamoDB</a> extensively for storing our data. We like it because it provides scalability and performance while being almost completely hands-off from an operational perspective.</p><p>AWS <a href="https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-transactions/">recently announced</a> support for transactions in DynamoDB. Transactions guarantee that a set of database operations are performed as an atomic unit, and are <a href="https://en.wikipedia.org/wiki/ACID_(computer_science)">ACID</a> compliant. In the case of an error during any part of the transaction, all operations will be rolled back, helping to guarantee data integrity and consistency across many rows or tables.</p><p>This is particularly useful for modelling relational data, where relationships between entities are written across multiple records and all need to be present to maintain integrity.</p><p>We’re going to run through an example of how one might leverage transactions when writing relational data in DynamoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/0*fenT4Ze1t2PrdeXT.jpg" /></figure><h3>Sample Relational Data Scenario</h3><p>Let’s take a simplified example of flights between airports. In this scenario we have three related entities:</p><ol><li>Airports</li><li>The routes or connections between airports</li><li>Airline companies that operate routes</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*zuvP3nH3ixbvvMWIgcdkaw.png" /><figcaption>Sample routes between three airports</figcaption></figure><p>When modelling relational data in a NoSQL database like DynamoDB, it is important to think about how you will want to access the data before you start. In our case, we want to organize the data in a way to answer the following questions:</p><ol><li>What routes does a given company operate?</li><li>What routes enter or leave airport A, or connect airport A and airport B?</li></ol><p>Using techniques described in the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-relational-modeling.html">Best Practices for Modeling Relational Data with DynamoDB</a>, we will start with the following rows in our table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*hJac_wGyVPMOhrgJIDxvrw.png" /><figcaption>Sample Data</figcaption></figure><p>We can then use standard and efficient DynamoDB queries to retrieve the data for the access patterns we defined.</p><blockquote>What routes does Company-A operate?</blockquote><p>Use the Global Secondary Index partition key (GSI-PK) to query on <em>Company-A. </em>Extract the route from the PK attribute to get “Route-1” and “Route-2”.</p><blockquote>What routes connect LAX and JFK?</blockquote><p>Use the primary key to query where PK=<em>Airport-LAX</em>, SK <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html">BEGINS_WITH</a> <em>Airport-JFK. </em>Extract the route from the SK attribute to get “Route-1” .</p><h3>Adding a new route into the system</h3><p>Let’s say that “<em>Undercutter Air” </em>is introducing a new flight direct from LAX to LFH.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*8TLsnor7l-l7hkAfj5iUiA.png" /><figcaption>Newly added route</figcaption></figure><p>To add this route into our system we need to insert three new rows to our table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*GQv-I-3ijcMUWsUWVtkxug.png" /><figcaption>Newly Added Rows</figcaption></figure><p>Historically, your best option to add these three rows as an atomic unit was to perform a <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html">batch write</a>. Using the AWS JavaScript SDK, the code would look something like this:</p><pre><strong>const params </strong>= {<br>    <strong>RequestItems</strong>: {<br>        <strong>&#39;Table-1&#39;</strong>: [{<br>            <strong>PutRequest</strong>: {<br>                <strong>Item</strong>: {<br>                    <strong>PK</strong>: <strong>&#39;Route-3&#39;</strong>,<br>                    <strong>SK</strong>: <strong>&#39;Company-B&#39;</strong>,<br>                    <strong>From</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                    <strong>To</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>                }<br>            }<br>        }, {<br>            <strong>PutRequest</strong>: {<br>                <strong>Item</strong>: {<br>                    <strong>PK</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                    <strong>SK</strong>: <strong>&#39;Airport-LHR|Route-3&#39;</strong>,<br>                }<br>            }<br>        }, {<br>            <strong>PutRequest</strong>: {<br>                <strong>Item</strong>: {<br>                    <strong>PK</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>                    <strong>SK</strong>: <strong>&#39;Airport-LAX|Route-3&#39;</strong>,<br>                }<br>            }<br>        }]<br>    }<br>};<br><br><strong>const documentClient </strong>= <strong>new </strong>AWS.DynamoDB.DocumentClient();<br><br><strong>documentClient</strong>.batchWrite(<strong>params</strong>, <strong>function </strong>(err, data) {<br>    <strong>if </strong>(err) <strong>console</strong>.log(err);<br>    <strong>else console</strong>.log(data);<br>});</pre><p>While this would work a vast majority of the time, there are cases where a portion of the batch write could fail, leaving your data in an inconsistent state. You can, and should, follow the recommendations of retrying any unprocessed keys until they succeed. But for how long do you retry? What happens if your system shuts down while you are retrying? What happens if somebody tries to query the data when only a portion of the keys are written? There are no simple solutions to these problems.</p><h3>Using DynamoDB Transactions</h3><p>To write the same data as an atomic transaction, we can instead use the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html">TransactWrite</a> call in the JavaScript SDK. The code would look something like this:</p><pre><strong>const params </strong>= {<br>    <strong>TransactItems</strong>: [{<br>        <strong>Put</strong>: {<br>            <strong>TableName </strong>: <strong>&#39;Table-1&#39;</strong>,<br>            <strong>Item</strong>: {<br>                <strong>PK</strong>: <strong>&#39;Route-3&#39;</strong>,<br>                <strong>SK</strong>: <strong>&#39;Company-B&#39;</strong>,<br>                <strong>From</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                <strong>To</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>            }<br>        }<br>    }, {<br>        <strong>Put</strong>: {<br>            <strong>TableName </strong>: <strong>&#39;Table-1&#39;</strong>,<br>            <strong>Item</strong>: {<br>                <strong>PK</strong>: <strong>&#39;Airport-LAX&#39;</strong>,<br>                <strong>SK</strong>: <strong>&#39;Airport-LHR|Route-3&#39;</strong>,<br>            }<br>        }<br>    }, {<br>        <strong>Put</strong>: {<br>            <strong>TableName </strong>: <strong>&#39;Table-1&#39;</strong>,<br>            <strong>Item</strong>: {<br>                <strong>PK</strong>: <strong>&#39;Airport-LHR&#39;</strong>,<br>                <strong>SK</strong>: <strong>&#39;Airport-LAX|Route-3&#39;</strong>,<br>            }<br>        }<br>    }]<br>};<br><br><strong>const documentClient </strong>= <strong>new </strong>AWS.DynamoDB.DocumentClient();<br><br><strong>documentClient</strong>.transactWrite(<strong>params</strong>, <strong>function</strong>(err, data) {<br>    <strong>if </strong>(err) <strong>console</strong>.log(err);<br>    <strong>else console</strong>.log(data);<br>});</pre><p>That’s it! It will succeed or fail as an atomic unit. No need for the added complexity of retries, or to worry about partial failures causing data integrity issues.</p><p>You can also perform transactions across tables, or perform deletes and updates in the same transaction, if your relational model requires it.</p><p><strong>Update: Note </strong><a href="https://aws.amazon.com/about-aws/whats-new/2019/06/amazon-dynamodb-now-supports-up-to-25-unique-items-and-4-mb-of-data-per-transactional-request/">as of June 2019</a> there is now an increased <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-dynamodb-transactions">limit of 25 operations</a> per transaction, so if you need to update more than 25 rows you may need to revert to batch writing or perform multiple transactions.</p><h3>References</h3><p>You can learn more about advanced design patterns in DynamoDB by watching <a href="https://www.youtube.com/watch?v=HaEPXoXVf2k">this great presentation from Re:Invent 2018</a>.</p><p>Learn more about modelling relational data from AWS best practices:</p><ul><li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-relational-modeling.html">Best practices for modeling relational data in DynamoDB</a></li><li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html">Best practices for managing many-to-many relationships in DynamoDB tables</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c13619ca58ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/using-dynamodb-transactions-to-model-relational-data-c13619ca58ae">Using DynamoDB transactions to model relational data</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time Aggregation with DynamoDB Streams]]></title>
            <link>https://medium.com/signiant-engineering/real-time-aggregation-with-dynamodb-streams-f93547cfb244?source=rss-98789c49782e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f93547cfb244</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <dc:creator><![CDATA[Peter Hodgkinson]]></dc:creator>
            <pubDate>Mon, 11 Jun 2018 16:58:25 GMT</pubDate>
            <atom:updated>2019-12-13T14:46:53.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gVDQ4sEfoKssi5km" /><figcaption>Photo by <a href="https://unsplash.com/@hharritt?utm_source=medium&amp;utm_medium=referral">Hunter Harritt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At Signiant we help our customers move their data quickly. Some of our customers transfer <em>a lot</em> of data. Terabytes upon terabytes, every month.</p><p>We also strive to give our customers insight into how they are using our product, and feedback on how much data they are moving.</p><p>Building a system to meet these two requirements leads to a typical problem in data-intensive applications:</p><blockquote>How do you collect and write a ton of data, but also provide an optimal way to read that same data? Can you build this system to be scalable? Resilient to errors? Is it easy to implement and operate?</blockquote><p>A typical solution to this problem would be to write a batch process for combining this mass of data into aggregated rows. This approach has a few inherent problems:</p><ul><li>You need to operate and monitor a fleet of servers to perform the batch operations. Often this comes in the form of a Hadoop cluster.</li><li>You need to schedule the batch process to occur at some future time. But what happens if you want to query the data before that time?</li><li>What happens when something goes wrong with the batch process? Do you know how to resume from the failure point? Have you lost any data? What does it mean for your application if the previous batch didn’t succeed?</li></ul><p>Is there a better way? Can you produce aggregated data in real-time<em>,</em> in a scalable way, without having to manage servers?</p><h3>Using DynamoDB Streams</h3><p>At Signiant we use AWS’s <a href="https://aws.amazon.com/dynamodb">DynamoDB</a> extensively for storing our data. We like it because it provides scalability and performance while being almost completely hands-off from an operational perspective.</p><p><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html">DynamoDB Streams</a> is a feature of DynamoDB that can send a series of database events to a downstream consumer. This consumer can be an application you write and manage yourself, or an AWS <a href="https://aws.amazon.com/lambda">Lambda</a> function you write and allow AWS to manage and trigger. Depending on the operation that was performed on your source table, your application will receive a corresponding INSERT, MODIFY, or REMOVE event. The event will also include a snapshot of the data contained in the database row before and after it was changed.</p><p>For example, if a new row gets written to your source table, the downstream application will receive an INSERT event that will look something like this:</p><pre>{  <br>   <strong>&quot;Records&quot;</strong>:[  <br>      {  <br>         <strong>&quot;eventID&quot;</strong>:&quot;7de3041dd709b024af6f29e4fa13d34c&quot;,<br>         <strong>&quot;eventName&quot;</strong>:&quot;INSERT&quot;,<br>         <strong>&quot;eventVersion&quot;</strong>:&quot;1.1&quot;,<br>         <strong>&quot;eventSource&quot;</strong>:&quot;aws:dynamodb&quot;,<br>         <strong>&quot;awsRegion&quot;</strong>:&quot;us-east-1&quot;,<br>         <strong>&quot;dynamodb&quot;</strong>:{  <br>            <strong>&quot;ApproximateCreationDateTime&quot;</strong>:1479499740,<br>            <strong>&quot;Keys&quot;</strong>:{  <br>               <strong>&quot;Timestamp&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;2016-11-18:12:09:36&quot;<br>               },<br>               <strong>&quot;Username&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;John Doe&quot;<br>               }<br>            },<br>            <strong>&quot;NewImage&quot;</strong>:{  <br>               <strong>&quot;Timestamp&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;2016-11-18:12:09:36&quot;<br>               },<br>               <strong>&quot;Message&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;This is a bark from the Woofer social network&quot;<br>               },<br>               <strong>&quot;Username&quot;</strong>:{  <br>                  <strong>&quot;S&quot;</strong>:&quot;John Doe&quot;<br>               }<br>            },<br>            <strong>&quot;SequenceNumber&quot;</strong>:&quot;13021600000000001596893679&quot;,<br>            <strong>&quot;SizeBytes&quot;</strong>:112,<br>            <strong>&quot;StreamViewType&quot;</strong>:&quot;NEW_IMAGE&quot;<br>         },<br>         <strong>&quot;eventSourceARN&quot;</strong>:&quot;arn:aws:dynamodb:us-east-1:123456789012:table/BarkTable/stream/2016-11-16T20:42:48.104&quot;<br>      }<br>   ]<br>}</pre><h3>Aggregating the data using AWS Lambda</h3><p><em>What if we use the data coming from these streams to produce aggregated data on-the-fly and leverage the power of AWS Lambda to scale-up seamlessly?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*K7ytEMpQCgVwgfhuc8ll3w.jpeg" /></figure><p>Presume we are writing records to a source DynamoDB table of the following schema:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dc9b046cd9451d3da5559d1f7f64ff8/href">https://medium.com/media/2dc9b046cd9451d3da5559d1f7f64ff8/href</a></iframe><p>If we want to produce a daily sum of all bytes transferred by a customer on a given day, our daily rollup table schema might look something like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39437a385eb4976c69ee38dacd9be693/href">https://medium.com/media/39437a385eb4976c69ee38dacd9be693/href</a></iframe><p>Given these two schemas, we want our system to take a set of rows from the source table that looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d834f3832ebccd2c198bfc34512b4bc5/href">https://medium.com/media/d834f3832ebccd2c198bfc34512b4bc5/href</a></iframe><p>And produce entries in the aggregated table that looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dec5c22bc2c4ead0d27ac921e466f11/href">https://medium.com/media/3dec5c22bc2c4ead0d27ac921e466f11/href</a></iframe><p>In the real world we write tens of thousands of rows into the source table per customer per day. It quickly becomes apparent that simply querying all the data from the source table and combining it on-demand is not going to be efficient. However querying a customer’s data from the daily aggregation table will be efficient for many years worth of data.</p><h4>The Lambda Function (Node 8.10)</h4><p><em>Note that the following assumes you have created the tables, enabled the DynamoDB stream with a Lambda trigger, and configured all the IAM policies correctly. You refer to </em><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.Tutorial.html"><em>this tutorial</em></a><em> for a quick overview of how to do all this.</em></p><pre><strong>const </strong>AWS = require(<strong>&#39;aws-sdk&#39;</strong>);<br><strong>const </strong>documentClient = <strong>new </strong>AWS.<strong>DynamoDB</strong>.<strong>DocumentClient</strong>({});<br><strong>async function </strong>handler(event, context, callback) {<br>    <strong>try </strong>{<br>        <em>//trim down to just &quot;INSERT&quot; events<br>        </em><strong>const </strong>insertRecords = event.<strong>Records</strong>.filter(<strong>record </strong>=&gt; record.eventName === <strong>&#39;INSERT&#39;</strong>);<br><em>// Unnmarshall records them to plain JSON objects<br>        </em><strong>const </strong>unmarshalledRecords = insertRecords.map(<strong>record </strong>=&gt;<br>            AWS.<strong>DynamoDB</strong>.<strong>Converter</strong>.unmarshall(record.<strong>dynamodb</strong>.NewImage)<br>    );<br>        <strong>for </strong>(let record of unmarshalledRecords) {<br>            <em>// this could be changed to any time window (a month, a year) and, in theory, the below algorithm would still work<br>            </em><strong>const </strong>MS_IN_DAY = 1000 * 60 * 60 * 24;<br>            <strong>const </strong>recordEndTime = <strong>new Date</strong>(record.TransferTime);<br>            <strong>const </strong>aggregatedDay = recordEndTime.getTime() - recordEndTime.getTime() % MS_IN_DAY;<br>            <strong>const </strong>params = {<br>                <strong>TableName</strong>: <strong>&#39;DailyAggregateTable&#39;</strong>,<br>                <strong>Key</strong>: {<br>                    <strong>CustomerId</strong>: record.CustomerId,<br>                    <strong>Day</strong>: <strong>new Date</strong>(aggregatedDay).toISOString(),<br>                },<br>                <strong>UpdateExpression</strong>: <strong>&#39;ADD #TotalBytes :Bytes, #CombinedRecords :CombinedRecords&#39;</strong>,<br>                <strong>ExpressionAttributeNames</strong>: {<br>                    <strong>&#39;#TotalBytes&#39;</strong>: <strong>&#39;TotalBytes&#39;</strong>,<br>                    <strong>&#39;#CombinedRecords&#39;</strong>: <strong>&#39;CombinedRecords&#39;</strong>,<br>                },<br>                <strong>ExpressionAttributeValues</strong>: {<br>                    <strong>&#39;:Bytes&#39;</strong>: record.Bytes,<br>                    <strong>&#39;:CombinedRecords&#39;</strong>: 1<br>                },<br>            };<br>            <strong>try </strong>{<br>                <em>//Write updates to daily rollup table<br>                </em>await documentClient.update(params).promise();<br>            } <strong>catch </strong>(err) {<br>                <em>//Swallow any errors<br>                </em><strong>console</strong>.error(<br>                    <strong>`Internal Error: Error updating dynamoDB record with keys [</strong>${<strong>JSON</strong>.stringify(<br>                        params.<strong>Key<br>                    </strong>)}<strong>] and Attributes [</strong>${<strong>JSON</strong>.stringify(params.<strong>ExpressionAttributeValues</strong>)}<strong>]`<br>                </strong>);<br>                <strong>console</strong>.error(err);<br>            }<br>        }<br>        callback(<strong>null</strong>, <strong>`Successfully processed </strong>${event.<strong>Records</strong>.<strong>length</strong>}<strong> records.`</strong>);<br>    } <strong>catch </strong>(err) {<br>        <strong>console</strong>.error(<strong>`Error processing records. Event was [</strong>${<strong>JSON</strong>.stringify(event)}<strong>`</strong>);<br>        <strong>console</strong>.error(err);<br><em>//Note we don&#39;t actually fail the lambda function here by calling back with the error e.g. callback(err)<br>        </em>callback(<strong>null</strong>, <strong>`Swallowed the error </strong>${<strong>JSON</strong>.stringify(err)}<strong>`</strong>);<br>    }<br>}<br>exports.handler = handler;</pre><p>A few snippets of interest</p><pre><strong>const </strong>insertRecords = event.<strong>Records</strong>.filter(record =&gt; record.eventName === <strong>&#39;INSERT&#39;</strong>);</pre><p>Here we are filtering the records down to just INSERT events. In theory you can just as easily handle DELETE events by removing data from your aggregated table or MODIFY events by calculating the difference between the old and new records and updating the table</p><pre><strong>UpdateExpression</strong>: <strong>&#39;ADD #TotalBytes :Bytes, #CombinedRecords :CombinedRecords&#39;</strong>,<br><strong>ExpressionAttributeNames</strong>: {<br>   <strong>&#39;#TotalBytes&#39;</strong>: <strong>&#39;TotalBytes&#39;</strong>,<br>   <strong>&#39;#CombinedRecords&#39;</strong>: <strong>&#39;CombinedRecords&#39;</strong>,<br>},<br><strong>ExpressionAttributeValues</strong>: {<br>   <strong>&#39;:Bytes&#39;</strong>: record.Bytes,<br>   <strong>&#39;:CombinedRecords&#39;</strong>: 1<br>},</pre><p>DynamoDB doesn’t support record-level locking, so how do you ensure that two lambda functions writing the same record at the same time they don’t both overwrite the initial value instead correctly aggregating both values? One answer is to use <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html">update expressions</a>. Here we are using an update expression to atomically add to the pre-existing <em>Bytes</em> value. Two, near-simultaneous, updates will successfully update the aggregated value without having to know the previous value.</p><p>There are a few different ways to use update expressions. For example, if you wanted to add a <em>createdOn</em> date that was written on the first update, but then not subsequently updated, you could add something like this to your expression:</p><pre><strong>SET #createdOn = if_not_exists(#createdOn, :now)</strong></pre><pre><strong>try </strong>{<br>   <em>//Write updates to daily rollup table<br>   </em><strong>await </strong>documentClient.update(params).promise();<br>} <strong>catch </strong>(err) {<br>   <em>//Swallow any errors<br>   </em><strong>console</strong>.error(err);<br>}</pre><pre>...</pre><pre><em>//Note we don&#39;t actually fail the lambda function here by calling back with the error e.g. callback(err)<br></em>callback(<strong>null</strong>, `Swallowed the error ${JSON.stringify(err)}`);</pre><p>Here we are swallowing any errors that occur in our function and not triggering the callback with an error. E.g. Not calling <em>callback(err)</em>. This is because your Lambda will get triggered with a batch of events in a single invocation (this can be changed by setting the <a href="https://docs.aws.amazon.com/lambda/latest/dg/API_CreateEventSourceMapping.html#API_CreateEventSourceMapping_RequestSyntax">BatchSize</a> property of the Lambda DynamoDB Stream event source), and you generally don’t want to fail the entire batch. This will be discussed more below.</p><h3>Gotchas and Lessons Learned</h3><h4>Kinesis, Batch Size, Error Handling, and Partial Failure</h4><p>Under the hood, DynamoDB uses <a href="https://aws.amazon.com/kinesis/data-streams/">Kinesis</a> to stream the database events to your consumer. By its nature, Kinesis just stores a log of events and doesn’t track how its consumers are reading those events. It simply provides an interface to fetch a number of events from a given point in time. It’s up to the consumer to track which events it has received and processed, and then request the next batch of events from where it left off (luckily AWS hides this complexity from you when you choose to connect the event stream to a Lambda function). This is a different paradigm than <a href="https://aws.amazon.com/sqs/">SQS</a>, for example, which ensures that <em>only one </em>consumer can process a given message, or set of messages, at a given time. In SQS you can then delete a single message from the queue so it does not get processed again. In Kinesis there is no concept of deleting an event from the log.</p><p>The inability to control the set of events that is coming from the stream introduces some challenges when dealing with errors in the Lambda function. <strong>There is no concept of a partial success</strong>. I.E. you can’t send information back to the stream saying: “I processed these 50 events successfully, and these 50 failed, so please retry the 50 that failed”. If you fail your entire Lambda function, the DynamoDB stream will resend the entire set of data again in the future. This is problematic if you have already written part of your data to the aggregate table. How do you prevent duplicate records from being written? And how do you handle incoming events that will never succeed, such as invalid data that causes your business logic to fail? There is no silver bullet solution for this case, but here are some ideas:</p><ul><li>Do some data-sanitization of the source events. If you can identify problems and throw them away before you process the event, then you can avoid failures down-the-line. Log the failures and possibly set up some CloudWatch Alarms to notify you of these unexpected cases. We used <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringPolicyExamples.html">CloudWatch Metric Filters</a> to translate our log errors into metrics we could observe and trigger alarms for.</li><li>Perform retries and backoffs when you encounter network or throughput exceptions writing to the aggregate table. This provides you more opportunity to succeed when you are approaching your throughput limits. If you are using an AWS SDK you get this <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.RetryAndBackoff">for free</a>.</li><li>If all else fails, write the event you are currently processing to some secondary storage. You cannot throw away this data if you want your destination table to be an accurate aggregate of the source table. Writing the event to an SQS queue, or S3, or even another table, allows you to have a second chance to process the event at later time, ideally after you have adjusted your throughput, or during a period of lighter usage. We implemented an SQS queue for this purpose.</li><li>Set your BatchSize to 1. I wouldn’t generally recommend this, as the ability to process and aggregate a number of events at once is a huge performance benefit, but it would work to ensure you aren’t losing data on failure. Simply trigger the Lambda callback with an error, and the failed event will be sent again on the next invocation. With this approach you have to ensure that you can handle events quickly enough that you don’t fall too far behind in processing the stream. A DynamoDB stream will only persist events for 24 hours and then you will start to lose data. You can monitor the <a href="https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html">IteratorAge</a> metrics of your Lambda function to determine how far behind you might be.</li></ul><h4>DynamoDB Throughput, Concurrency, Partitions, and Batch Writes</h4><p>Although DynamoDB is mostly hands-off operationally, one thing you <em>do </em>have to manage is your read and write throughput limits. Setting these to the correct values is an inexact science. Set them too low and you start getting throughput exceptions when trying to read or write to the table. Set them too high and you will be paying for throughput you aren’t using. <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html">Auto-scaling</a> can help, but won’t work well if you tend to read or write in bursts, and there’s still no guarantee you will never exceed your throughput limit.</p><p>In our scenario we specifically care about the write throughput on our aggregate table. We want to allow our Lambda function to successfully write to the aggregate rows without encountering a throughput exception. The logical answer would be to set the write throughput on the aggregate table to the same values as on the source table. After all, a single write to the source table should equate to a single update on the aggregate table, right? Unfortunately, the answer is a little more complicated than that.</p><p>First, you have to consider the number of Lambda functions which could be running in <em>parallel</em>. E.g. if you are running two Lambdas in parallel you will need double the throughput that you would need for running a single instance. The potential number of Lambdas that could be triggered in parallel for a given source table is actually based on the number of database <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.Partitions.html"><em>partitions</em></a><em> </em>for that table. There is one stream per partition. Unfortunately there is no concrete way of knowing the exact number of partitions into which your table will be split. It is a factor of the total provisioned throughput on the table and the amount of data stored in the table that roughly works out to something like</p><pre>Total partitions = <br>  MAX(Total partitions for desired performance, <br>      Total partitions for desired capacity)</pre><p>See this <a href="https://shinesolutions.com/2016/06/27/a-deep-dive-into-dynamodb-partitions/">article</a> for a deeper dive into DynamoDB partitions. You can get a rough idea of how many Lambda functions are running in parallel by looking at the number of separate CloudWatch logs your function is generating at any given time. There should be about one per partition assuming you are writing enough data to trigger the streams across all partitions.</p><p>You can also manually control the <a href="https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html">maximum concurrency</a> of your Lambda function. For example, if you tend to write a lot of data in bursts, you could set the maximum concurrency to a lower value to ensure a more predictable write throughput on your aggregate table. Again, you have to be careful that you aren’t falling too far behind in processing the stream, otherwise you will start to lose data.</p><p>Secondly, if you are writing to the source table in batches using the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html">batch write</a> functionality, you have to consider how this will affect the number of updates to your aggregate table. For example, a batch write call can write up to 25 records at a time to the source table, which could conceivably consume just 1 unit of write throughput. This will translate into 25 separate INSERT events on your stream. Since updating an item with update expressions cannot be done in batches, you will need to have 25x the throughput on the destination table to handle this case.</p><p>There is opportunity for optimization, such as combining the batch of events in memory in the Lambda function, where possible, before writing to the aggregate table. In practice, we found that having the write throughput on the aggregate table set to twice that of the source comfortably ensures we will not exceed our limits, but I would encourage you to monitor your usage patterns to find the number that works for your case.</p><h3>In conclusion</h3><p>Using the power of DynamoDB Streams and Lambda functions provides an easy to implement and scalable solution for generating real-time data aggregations. As a bonus, there is little to no operational overhead. The pattern can easily be adapted to perform aggregations on different bucket sizes (monthly or yearly aggregations), or with different properties, or with your own conditional logic. You could even configure a separate stream on the aggregated daily table and chain together multiple event streams that start from a single source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*oQ_h-g3FM--N0uj8.jpg" /></figure><p>There are a few things to be careful about when using Lambda to consume the event stream, especially when handling errors. Understanding the underlying technology behind DynamoDB and Kinesis will help you to make the right decisions and ensure you have a fault-tolerant system that provides you with accurate results.</p><p>Happy streaming!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f93547cfb244" width="1" height="1" alt=""><hr><p><a href="https://medium.com/signiant-engineering/real-time-aggregation-with-dynamodb-streams-f93547cfb244">Real-Time Aggregation with DynamoDB Streams</a> was originally published in <a href="https://medium.com/signiant-engineering">Signiant Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>