<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[circuitpeople - Medium]]></title>
        <description><![CDATA[Ramblings related to the design and engineering of things, the people that engage in them, and other such interests. - Medium]]></description>
        <link>https://medium.com/circuitpeople?source=rss----ab6443916890---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>circuitpeople - Medium</title>
            <link>https://medium.com/circuitpeople?source=rss----ab6443916890---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:25:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/circuitpeople" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stream-to-Stream S3 Uploads with AWS Lambda]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/circuitpeople/stream-to-stream-s3-uploads-with-aws-lambda-578fe710ac1e?source=rss----ab6443916890---4"><img src="https://cdn-images-1.medium.com/max/1043/1*aliBE9EjoOtwY01tpM-qHw.png" width="1043"></a></p><p class="medium-feed-snippet">Transforming the content of large S3 objects with Lambda seems daunting, but a simple stream adapter makes some amazing things possible&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/circuitpeople/stream-to-stream-s3-uploads-with-aws-lambda-578fe710ac1e?source=rss----ab6443916890---4">Continue reading on circuitpeople »</a></p></div>]]></description>
            <link>https://medium.com/circuitpeople/stream-to-stream-s3-uploads-with-aws-lambda-578fe710ac1e?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/578fe710ac1e</guid>
            <category><![CDATA[net-core]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[stream]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Sun, 17 Jan 2021 05:19:04 GMT</pubDate>
            <atom:updated>2021-01-17T05:13:30.617Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Random-Access (Seekable) Streams for Amazon S3 in C#]]></title>
            <link>https://medium.com/circuitpeople/random-access-seekable-streams-for-amazon-s3-in-c-bd2414255dcd?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/bd2414255dcd</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Tue, 10 Nov 2020 22:52:41 GMT</pubDate>
            <atom:updated>2020-11-15T00:41:18.380Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BmVYi5qmBdjYTtPzCQhFA.jpeg" /><figcaption>Some stream interfaces to S3 objects are better than others.</figcaption></figure><p>Some files are big enough that working on them in memory isn’t desirable, or even possible. Even if memory size isn’t an issue, the data transfer moving the entire file from S3 can be expensive and wasteful.</p><p>In this article I demonstrate using the S3 API’s to implement optimized reads via a general-purpose, “seekable” Stream implementation in C#. This stream can be used with almost any existing code library or package, and I give examples using DiscUtils, MetadataExtractor, Parquet.Net and the .Net Core compression library.</p><p>The solution is shown to provide orders of magnitude reduction in network traffic when performing simple operations on ISO, Image, Zip and Parquet files and performs similarly with other complex binary formats up to the 5TB S3 object size limit without noticeable degradation in performance (so long as only a small amount of the content is actually being used).</p><h3>Trial and Error</h3><p>Let’s set the stage: Suppose I have disc image files in ISO format on Amazon S3 from which I need to extract some information from just one or two files. I will only do this on-demand (e.g. when someone clicks a button) and can’t afford to spend much money on it.</p><p>I’m familiar with the excellent <a href="https://github.com/DiscUtils/DiscUtils">DiscUtils</a> library, and know I want to use it since the ISO format is complicated and I don’t want to invent the wheel to read from them:</p><pre>dotnet new console<br>dotnet add package AWSSDK.S3<br>dotnet add package DiscUtils</pre><p>The DiscUtils website example code for opening a disk image and reading a file looks like it’s very straightforward, and it seems like it will be simple to grab the stream from S3 and use that to hand to the CDReader:</p><pre>using (FileStream isoStream = File.Open(@&quot;C:\temp\sample.iso&quot;))<br>{<br> CDReader cd = new CDReader(isoStream, true);<br> Stream fileStream = cd.OpenFile(@&quot;Folder\Hello.txt&quot;, FileMode.Open);<br> // Use fileStream...<br>}</pre><p>That being the case, the code for reading the ISO from S3 would be something simple like this:</p><pre>var s3 = new AmazonS3Client();<br>using var response = await s3.GetObjectAsync(BUCKET, KEY);<br>using var iso = <strong>new CDReader(response.ResponseStream, true)</strong>;<br>using var file = iso.OpenFile(FILENAME, FileMode.Open, FileAccess.Read);<br>using var reader = new StreamReader(file);<br>var content = await reader.ReadToEndAsync();<br>await Console.Out.WriteLineAsync(content);</pre><p>Most libraries that work with files by taking filesystem paths will also be able to work with Stream objects (not all, but most). DiscUtils does, as the CDReader class may be constructed with either. The ZipArchive class in System.IO.Compression is another example.</p><p>So the code above should work, right? Nope.</p><pre>Unhandled exception. System.NotSupportedException: <strong>Specified method is not supported</strong>.<br> at System.Net.Http.HttpBaseStream.<strong>set_Position</strong>(Int64 value)<br> at DiscUtils.Iso9660.VfsCDReader..ctor(Stream data, Iso9660Variant[] variantPriorities, Boolean hideVersions)<br> at DiscUtils.Iso9660.VfsCDReader..ctor(Stream data, Boolean joliet, Boolean hideVersions)<br> at DiscUtils.Iso9660.CDReader..ctor(Stream data, Boolean joliet)<br> at Seekable_S3_Stream.Program.Main(String[] args) in C:\Users\lee.PC\Seekable S3 Stream\Program.cs:line 17<br> at Seekable_S3_Stream.Program.&lt;Main&gt;(String[] args)</pre><p>Our first attempt throws a NotSupportedException when run, specifically in the HTTP stream class. It makes sense that a library to read ISO files would seek to positions in the file because usually it can (files read from disk support seek operations) and because the format of ISOs encourages it by making extensive use of dictionaries and file position offsets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*gaBw4RH9Mn7I9_4Yn-yfpQ.png" /><figcaption>Like ISO files, Zip files are structured with a “central directory” at the end which “points” to file content within the file with relative offsets. To find a single file, the directory needs to be read first, then the section of the file containing the desired content. Reading any other part of the file is unnecessary.</figcaption></figure><p>It also makes sense that an HTTP stream isn’t “seekable”, but why maybe isn’t obvious. The HTTP protocol doesn’t include a way to stop, rewind and replay a download. Well, that’s not quite right — HTTP *does* define a way to do that, but it’s optional and not all (few in fact) web servers support it. Lucky for us, S3 is one of those HTTP services that does support <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests">HTTP’s method</a> for “seeking” by using Range headers (which I’ve written about <a href="https://medium.com/circuitpeople/serverless-large-file-downloads-to-s3-a11b4ef4788e">before</a>). It’s not quite the same, though, so we’re going to need to get clever to make this work.</p><p>On <a href="https://stackoverflow.com/questions/3434007/error-this-stream-does-not-support-seek-operations-in-c-sharp">StackOverflow</a> you’ll find plenty of advice to workaround this issue by copying the S3 response stream into a MemoryStream. Let’s try that:</p><pre>…<br>using var response = await s3.GetObjectAsync(BUCKET, KEY);<br><strong>var ms = new MemoryStream();<br>await response.ResponseStream.CopyToAsync(ms);<br>ms.Seek(0, SeekOrigin.Begin);<br></strong>using var iso = new CDReader(ms, true);<br>…</pre><p>Now, that’ll do it for sure, right? Well, yes, sort of.</p><p>It works, albeit slowly. Very slowly, because the ISO files I’m working with are quite large. The available network bandwidth for a single Lambda execution is in the range of 1–2Gbps so the response with this approach will be so slow as to be unworkable. Moreover, fanout techniques won’t improve the effectiveness here. So, what to do? Punt and use a high-memory instance or container? Nope. Fix the <em>root problem</em>.</p><p>We need a Stream implementation that supports seeking to skip past sections of the file, or rewind to access previously read sections. That’s how the CDReader class expects to work with a stream so it only reads a tiny fraction of the file’s content to get at a single file. We can confirm this by creating a simple stream wrapper that records the number of bytes actually read (versus skipped by seeking from place to place):</p><pre>class ReadCountingStream : MemoryStream {<br> public long Total = 0L;<br> public override int Read(byte[] buffer, int offset, int count)<br> {<br> var read = base.Read(buffer, offset, count);<br> Total += read;<br> return read;<br> }<br>}<br>…<br> var ms = new ReadCountingStream();<br>…<br> await Console.Out.WriteLineAsync($&quot;{ms.Total/(float)ms.Length * 100}% read&quot;);<br>…</pre><p>Running the above code I gives me the percentage of the file that is actually used, and establish the potential for optimizing the performance of the code (the time to read the contents is the vast majority of the time used):</p><pre>0.003969788% read</pre><p>Well, that’s good news: Less that 1/100th of a percent of the file is actually read. If we implement a “perfect” stream that reads only those bytes from S3 we can expect an enormous performance improvement (probably bounded by latency rather than bandwidth). Cool, let’s do it.</p><p>This time instead of wrapping a concrete stream we’re going to create a ground-up implementation that makes calls to S3 when blocks of the file are needed based on the value of the Position property and number of bytes needed. The source for my SeekableS3Streamclass can be found on Github (link below), but here is how it’s used:</p><pre>…<br><strong>var stream = new Cppl.Utilities.AWS.SeekableS3Stream(s3, BUCKET, KEY, 1 * 1024 * 1024, 4);<br>using var iso = new CDReader(stream, true);<br></strong>using var file = iso.OpenFile(FILENAME, FileMode.Open, FileAccess.Read);<br>using var reader = new StreamReader(file);<br>var content = await reader.ReadToEndAsync();<br>…</pre><p>It’s even simpler than using S3 directly. To see how it performs we’ll again print the read statistics for comparison:</p><pre>await Console.Out.WriteLineAsync($&quot;{stream.TotalRead / (float)stream.Length * 100}% read, {stream.TotalLoaded / (float)stream.Length * 100}% loaded&quot;);</pre><p>Which prints:</p><pre>0.003969788% read, 0.3565448% loaded</pre><p>So, with the new stream in use the amount of data transferred from S3 drops by more than 99.6% — more than 100 times faster! And that’s before optimizing the size of the ranges being pulled, or the number held in the MRU list, so further improvement are probably possible.</p><p>This stream implementation isn’t perfect but it’s far, var better than the MemoryStream approach. So much so that it’s a no-brainer to drop that bad idea like a hot rock.</p><h3>Optimization</h3><p>Somewhere between a “pure” streaming solution where one buffer at a time is available and the “read it all” approach of a MemoryStream is the “right-sized” solution that pulls only the needed bytes from S3, and uses just enough memory to avoid repeated calls to S3 when the same file ranges are used multiple times.</p><p>Ideally, the right amount of memory is determined by the size of the Lambda instance (memory ranging from 256MB to 3GB) or container on which the code runs as well as the workload (amount of the file ultimately used). My SeekableS3Stream class lets me tweak the size of ranges and how many to hold in the recently used list. Those two parameters allow me to tune the performance for any given workload.</p><p>The size of the ranges is important as it controls the “excess” bytes. If the ranges are too large for the workload, much of them will remain unused. For example, if the range size is 1MB and only the first KB is used that would potentially inefficient. On the flip side, if the range size is too small it can result in making a large number of calls to S3.</p><p>The number of ranges to hold in the MRU becomes important when the workload would result in repeated reads of some ranges. For example, Zip files contain a central directory structure</p><h3>Summary</h3><p>So if you’re working with very small files, an approach using byte[] or perhaps a MemoryStream provides effective read/write, random-access memory. But it’s sub-optimal from a cost perspective and won’t scale. Maybe consider using the seekable stream approach.</p><p>You can find my code for SeekableS3Stream as well as example for Parquet and other formats on <a href="https://github.com/mlhpdx/seekable-s3-stream">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd2414255dcd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/random-access-seekable-streams-for-amazon-s3-in-c-bd2414255dcd">Random-Access (Seekable) Streams for Amazon S3 in C#</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS SQS in Pictures]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/circuitpeople/aws-sqs-in-pictures-448c897534e1?source=rss----ab6443916890---4"><img src="https://cdn-images-1.medium.com/max/695/1*pMiA3AskSeErCZ4NRiboUw.png" width="695"></a></p><p class="medium-feed-snippet">Or, the simple complexity of SQS explained with sequence diagrams.</p><p class="medium-feed-link"><a href="https://medium.com/circuitpeople/aws-sqs-in-pictures-448c897534e1?source=rss----ab6443916890---4">Continue reading on circuitpeople »</a></p></div>]]></description>
            <link>https://medium.com/circuitpeople/aws-sqs-in-pictures-448c897534e1?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/448c897534e1</guid>
            <category><![CDATA[queue]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sequence-diagrams]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Tue, 31 Mar 2020 00:03:56 GMT</pubDate>
            <atom:updated>2020-03-31T19:29:02.742Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[AWS CLI with jq and Bash]]></title>
            <link>https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/9d54e2eabaf1</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[json]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Mon, 29 Oct 2018 19:14:01 GMT</pubDate>
            <atom:updated>2025-04-29T18:15:24.421Z</atom:updated>
            <content:encoded><![CDATA[<h4>The CLI is utilitarian, but a little jq sauce makes it beautiful</h4><p>Transitioning from using the AWS console UI to the command line isn’t easy. The CLI is holds the same power as the APIs, and the dump trucks of JSON. When working in code that isn’t a problem, but it is a little painful when presented as the response to CLI commands.</p><p>That’s where jq shines — taking globs of JSON data and filtering summarizing it to get to the good stuff. And it gets even better under Bash (sorry Powershell fans, I’m a Windows user but Windows is so much better with WSL).</p><p>Many of my interactions with the CLI begin with a question, so I’ve compiled the list below as examples of how to make the best of this powerful combination of tools. Note that I’m not using the--profile or --region options, which you may want to add. I hope you find some of them useful!</p><h4>When was my AWS user account created?</h4><p>Silly, but sometimes that’s justification enough:</p><pre>aws iam get-user | jq -r &quot;.User.CreateDate[:4]&quot;</pre><p>FYI, if you get 2010 or earlier, consider using something other than your root credentials (doh!). IAM was <a href="https://aws.amazon.com/about-aws/whats-new/2010/09/02/announcing-aws-identity-and-access-management-iam-preview-beta/">introduced as a beta</a> in September of 2010, so you may be “doing it wrong”.</p><h4>Which Account am I using?</h4><p>Maybe I’m the only one, but given the multitude of ways to specify the profile and/or keys to use with the CLI, and the plethora of accounts I work with in a given day, sometimes I really don’t know. This helps by combining the account identity, account number and account aliases in one nice little object:</p><pre>{ aws sts get-caller-identity &amp; aws iam list-account-aliases; } | jq -s &quot;.|add&quot;</pre><p>Yes, curly brackets for the win!</p><p>So, now that we know which account we’re in, which region will the CLI use? Given all the configuration options it’s surprisingly difficult to tell, so here’s a little command that tells you for certain:</p><pre>aws ec2 describe-availability-zones --output text --query &#39;AvailabilityZones[0].[RegionName]&#39;</pre><h4>Which Services am I using?</h4><p>This question comes up during security reviews a lot, and while it *can* be answered in the Config console UI (given enough clicks), or using Cost Explorer (fewer clicks), this is much faster and easy to automate:</p><pre>aws ce get-cost-and-usage --time-period Start=$(date &quot;+%Y-%m-01&quot; -d &quot;-1 Month&quot;),End=$(date --date=&quot;$(date +&#39;%Y-%m-01&#39;) - 1 second&quot; -I) --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq &#39;.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount &gt; 0) | .Keys[0]&#39;</pre><p>Bash (well, the date command) provides the dates for the previous month, but obviously you can make it what makes sense to you (perhaps the current month). That command will produce output like this:</p><pre>[<br>  &quot;AWS Budgets&quot;,<br>  &quot;AWS CloudTrail&quot;,<br>  &quot;AWS CodeCommit&quot;,<br>  &quot;AWS Config&quot;, <br>  <em>...snipped for length...</em><br>  &quot;Amazon WorkSpaces&quot;,<br>  &quot;Amazon WorkSpaces Application Manager&quot;,<br>  &quot;AmazonCloudWatch&quot;,<br>  &quot;CodeBuild&quot;<br>]</pre><h4>What is each service costing me?</h4><p>The AWS services you’re using can be different from the ones costing you the most. It’s easy enough to keep an eye on the high-level with this command (this time for the current month):</p><pre>aws ce get-cost-and-usage --time-period Start=$(date &quot;+%Y-%m-01&quot;),End=$(date --date=&quot;$(date +&#39;%Y-%m-01&#39;) + 1 month  - 1 second&quot; -I) --granularity MONTHLY --metrics USAGE_QUANTITY BLENDED_COST  --group-by Type=DIMENSION,Key=SERVICE | jq &#39;[ .ResultsByTime[].Groups[] | select(.Metrics.BlendedCost.Amount &gt; &quot;0&quot;) | { (.Keys[0]): .Metrics.BlendedCost } ] | sort_by(.Amount) | add&#39;</pre><p>I can’t guarantee you’ll like the numbers.</p><h4>How many instances of each type do I have, and in what states?</h4><p>Considering buying reserved instances or thinking about migrating to a newly introduced class?</p><pre>aws ec2 describe-instances | jq -r &quot;[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]&quot;</pre><p>Here is some sample output:</p><pre>[<br>  {<br>    &quot;state&quot;: &quot;running&quot;,<br>    &quot;types&quot;: [<br>      {<br>        &quot;type&quot;: &quot;x1e.8xlarge&quot;,<br>        &quot;count&quot;: 3<br>      },<br>      {<br>        &quot;type&quot;: &quot;m4.xlarge&quot;,<br>        &quot;count&quot;: 1<br>      },<br>      {<br>        &quot;type&quot;: &quot;t2.medium&quot;,<br>        &quot;count&quot;: 1<br>      },<br>      {<br>        &quot;type&quot;: &quot;t2.micro&quot;,<br>        &quot;count&quot;: 112<br>      }<br>    ]<br>  },<br>  {<br>    &quot;state&quot;: &quot;stopped&quot;,<br>    &quot;types&quot;: [<br>      {<br>        &quot;type&quot;: &quot;m4.large&quot;,<br>        &quot;count&quot;: 1<br>      }<br>    ]<br>  }<br>]</pre><p>So much easier to see this big picture this way. If you prefer the “flat world” view, this version will get the results in CSV:</p><pre>aws ec2 describe-instances | jq -r &quot;[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.)|.[]|{type: .[0].type, state: .[0].state, count: ([.[]]|length) }|[.type, .state, .count]|@csv&quot;</pre><h4>What CIDRs have Ingress Access to which Ports?</h4><p>This is helpful when you need to perform a survey or audit of your system boundaries. While such a task isn’t ever “easy”, it can go more smoothly with with a summary:</p><pre>aws ec2 describe-security-groups | jq &#39;[ .SecurityGroups[].IpPermissions[] as $a | { &quot;ports&quot;: [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, &quot;cidr&quot;: $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join(&quot;-&quot;)]|unique }] | add&#39;</pre><p>Hopefully you don’t see too many 0.0.0.0/0’s or 0–65535’s in there…</p><h4>Can I get a summary of my VPCs in CSV format?</h4><p>Speaking of CIDRs, this one is inspired by a comment from <a href="https://medium.com/u/2e6956ba9">Bruce Richardson</a>. If you want a quick overview of the VPCs in an account, and need it in CSV with headers this one is for you:</p><pre>aws ec2 describe-vpcs | jq -r &#39;[.Vpcs[] | {VpcId: .SubnetId, Name: (.Tags[]?|select(.Key==&quot;Name&quot;)|.Value), CIDR: .CidrBlock, State: .State, VpcId: .VpcId, AWSAccount: .OwnerId}]|(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ]])[]|@csv&#39;</pre><p>If you need that list for all regions, it’s a little more complicated (and slow) but easier than clicking-around the console:</p><pre>for region in $(aws ec2 describe-regions --query Regions[].RegionName --output text); do aws ec2 describe-vpcs --region $region | jq  &#39;.Vpcs[]|{Region: &quot;&#39;&quot;${region}&quot;&#39;&quot;, VpcId: .SubnetId, Name: (.Tags[]?|select(.Key==&quot;Name&quot;)|.Value), CIDR: .CidrBlock, State: .State, VpcId: .VpcId, AWSAccount: .OwnerId}&#39;; done | jq -s -r &#39;.|(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ]])[]|@csv&#39;</pre><h4>How do I Create a CodeCommit Repository and Clone It?</h4><pre>$ export REPO_URL=$(aws codecommit create-repository --repository-name <strong>&lt;name&gt;</strong> | jq -r &quot;.repositoryMetadata.cloneUrlHttp&quot;)<br>$ git clone $REPO_URL <strong>&lt;name&gt;</strong> &amp;&amp; cd <strong>&lt;name&gt;</strong></pre><p>Since the name repeats, this one is a better as an alias:</p><pre>.bashrc:<br>function make-and-clone-code-commit-repo()<br>{<br>export REPO_URL=$(aws codecommit create-repository --repository-name $1 | jq -r &quot;.repositoryMetadata.cloneUrlHttp&quot;)<br>git clone $REPO_URL $1 &amp;&amp; cd $1<br>}<br>alias mcc=&quot;make-and-clone-code-commit-repo&quot;</pre><p>Which makes it much easier to use:</p><pre>$ mcc <strong>&lt;name&gt;</strong></pre><h4>Which Lambda Functions Runtimes am I Using?</h4><p>And, are they all using the current runtime version or is someone going rogue?</p><pre>aws lambda list-functions | jq &quot;.Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] }<br>]&quot;</pre><p>Is everyone taking the time to set memory size and the time out appropriately?</p><pre>aws lambda list-functions | jq &quot;.Functions | group_by(.Runtime)|[.[]|{ (.[0].Runtime): [.[]|{ name: .FunctionName, timeout: .Timeout, memory: .MemorySize }] }]&quot;</pre><h4>Lambda Function Environment Variables</h4><p>This is a simple one, but helpful. Are you exposing secrets in variables? Have a typo in a key?</p><pre>aws lambda list-functions | jq -r &#39;[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length &gt; 0)&#39;</pre><h4>Creating EC2 Instances…</h4><p>Step 1: Find the right AMI (this is slow, ’cause there are a *lot* of AMIs) and hold it in an environment variable:</p><pre>export AMI_ID=$(aws ec2 describe-images --owners amazon | jq -r &quot;.Images[] | { id: .ImageId, desc: .Description } | select(.desc?) | select(.desc | contains(\&quot;Amazon Linux 2\&quot;)) | select(.desc | contains(\&quot;.NET 6\&quot;)) | .id&quot;)</pre><p>Step 2: Create a key pair, and hold on to it in a file:</p><pre>aws ec2 create-key-pair --key-name aurora-test-keypair &gt; keypair.pem</pre><p>Step 3: Create the instance using the AMI and the key pair, and hold onto the result in a file:</p><pre>aws ec2 run-instances --instance-type t2.micro --image-id $AMI_ID --region us-east-1 --subnet-id &lt;your_subnet_id&gt; --key-name keypair --count 1 &gt; instance.json</pre><p>Step 4: Grab the instance Id from the file:</p><pre>export INSTANCE_ID=$(jq -r .Instances[].InstanceId instance.json)</pre><p>Step 5: Wait for the instance to spin-up, then grab it’s IP address and hold onto it in an environment variable:</p><pre>export INSTANCE_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --output text --query &#39;Reservations[*].Instances[*].PublicIpAddress&#39;)</pre><p>Step 6: SSH and profit:</p><pre>ssh -i keypair.pem ec2-user@$INSTANCE_IP</pre><h4>What are my RDS Instance Endpoints?</h4><pre>aws rds describe-db-instances | jq -r &#39;.DBInstances[] | { (.DBInstanceIdentifier):(.Endpoint.Address + &quot;:&quot; + (.Endpoint.Port|tostring))}&#39;</pre><h4>How Many Services does AWS Have?</h4><p>Okay, not strictly a CLI thing but still a fun number to track.</p><pre>curl -s https://raw.githubusercontent.com/boto/botocore/develop/botocore/data/endpoints.json | jq -r &#39;.partitions[0].services | keys[]&#39; | wc -<br>l</pre><h4>How Many CloudFormation Stacks do I have in each Status?</h4><pre>aws cloudformation list-stacks | jq  &#39;.StackSummaries | [ group_by(.StackStatus)[] | { &quot;status&quot;: .[0].StackStatus, &quot;count&quot;: (. | length) }<br>]&#39;</pre><h4>Which EC2 Instances were created by Stacks?</h4><p>Hopefully none, but if you have them you know how important it is to be aware of their parentage.</p><pre>for stack in $(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE | jq -r &#39;.StackSummaries[].StackName&#39;); do aws cloudformation describe-stack-resources --stack-name $stack | jq -r &#39;.StackResources[] | select (.ResourceType==&quot;AWS::EC2::Instance&quot;)|.PhysicalResourceId&#39;; done;</pre><h4>How Many Gigabytes of Volumes do I have, by Status?</h4><pre>aws ec2 describe-volumes | jq -r &#39;.Volumes | [ group_by(.State)[] | { (.[0].State): ([.[].Size] | add) } ] | add&#39;</pre><p>Yeah. Maybe those should be cleaned up. ;)</p><h4>How many Snapshots do I have?</h4><p>Perhaps related to the number of volumes. Beware that the list is very, very, very long if you look at the ones owned by amazon.</p><pre>aws ec2 describe-snapshots --owner-ids self | jq &#39;.Snapshots | length&#39;</pre><p>And, how large are they in total?</p><pre>aws ec2 describe-snapshots --owner-ids self | jq &#39;[.Snapshots[].VolumeSize] | add&#39;</pre><p>And, how do they breakdown by the volume used to create them?</p><pre>aws ec2 describe-snapshots --owner-ids self | jq &#39;.Snapshots | [ group_by(.VolumeId)[] | { (.[0].VolumeId): { &quot;count&quot;: (.[] | length), &quot;size&quot;: ([.[].VolumeSize] | add) } } ] | add&#39;</pre><p>Note, when a Snapshot is copied the VolumeId of the copy does not reflect the volume of the original (it gets the special value vol-ffffffff).</p><h4>What’s happening in my Log Streams?</h4><p>It’d be cool if the CLI allowed me to get log events using the log stream ARN, but it doesn’t so it starts with getting the log group names:</p><pre>logs=$(aws logs describe-log-groups | jq -r &#39;.logGroups[].logGroupName&#39;)</pre><p>And, then maybe the first log stream for each:</p><pre>for group in $logs; do echo $(aws logs describe-log-streams --log-group-name $group --order-by LastEventTime --descending --max-items 1 | jq -r &#39;.logStreams[0].logStreamName + &quot; &quot;&#39;); done</pre><p>Or, loop through the groups and streams and get the last 10 messages since midnight:</p><pre>for group in $logs; do for stream in $(aws logs describe-log-streams --log-group-name $group --order-by LastEventTime --descending --max-items 1 | jq -r &#39;[ .logStreams[0].logStreamName + &quot; &quot;] | add&#39;); do echo &quot;&gt;&gt;&gt;&quot;; echo GROUP: $group; echo STREAM: $stream; aws logs get-log-events --limit 10 --log-group-name $group --log-stream-name $stream --start-time $(date -d &#39;today 00:00:00&#39; &#39;+%s%N&#39; | cut -b1-13) | jq -r &quot;.events[].message&quot;; done; done</pre><p>Finally, since log groups default retention period is “Never Expire” they can start to build up after a few years. I don’t use CloudWatch logs for long-term storage (and neither should you) but since AWS doesn’t provide a way to set the default retention to something I’d prefer, I run the following command from time to time to make sure they’re all set to 30 days:</p><pre>for group in $(aws logs describe-log-groups --query &quot;logGroups[].[logGroupName]&quot; --output text --no-paginate); do aws logs put-retention-policy --log-group-name $group --retention-in-days 30; done;</pre><h4>What logs does my Lambda Function generate when I run it?</h4><p>Simple, but necessary since Lambda base64 encodes the logs in the response.</p><p>In jq version 1.5 (you’ll get 1.4 unless you go out of your way to get 1.5 since it’s still in RC at this time):</p><pre>aws lambda invoke --function-name <strong>&lt;function name&gt;</strong> --payload &#39;{}&#39; --log-type Tail - | jq -r &#39;{ &quot;StatusCode&quot;: .StatusCode, &quot;LogResult&quot;: (.LogResult|@base64d)}&#39;</pre><p>In 1.4, you have to just rely on bash:</p><pre>aws lambda invoke --function-name <strong>&lt;function name&gt;</strong> --payload &#39;{}&#39; --log-type Tail - | jq -r &#39;.LogResult&#39; | base64 --decode</pre><h4>How Much Data is in Each of my Buckets?</h4><p>It’s remarkably difficult get the breakdown of data volume by bucket in the AWS console. CloudWatch contains the data, but if your account has more than a few buckets it’s very tedious to use.</p><p>This little command gives your the total size of the objects in each bucket, one per line, with human-friendly numbers:</p><pre>for bucket in $(aws s3api list-buckets --query &quot;Buckets[].Name&quot; --output text); do aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=$bucket Name=StorageType,Value=StandardStorage --start-time $(date --iso-8601)T00:00 --end-time $(date --iso-8601)T23:59 --period 86400 --statistic Maximum | echo $bucket: $(numfmt --to si $(jq -r &quot;.Datapoints[0].Maximum // 0&quot;)); done;</pre><p>Prefer to have that is dollars per month? Just a little math ( based on the current standard tier price of $0.023 per GB per month):</p><pre>for bucket in $(aws s3api list-buckets --query &quot;Buckets[].Name&quot; --output text); do aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=$bucket Name=StorageType,Value=StandardStorage --start-time $(date --iso-8601)T00:00 --end-time $(date --iso-8601)T23:59 --period 86400 --statistic Maximum | echo $bucket: \$$(jq -r &quot;(.Datapoints[0].Maximum //<br> 0) * .023 / (1024*1024*1024) * 100.0 | floor / 100.0&quot;); done;</pre><p>Warning, this one is slow.</p><h4>What do my Bucket Policies Allow?</h4><p>Nobody wants a <a href="https://www.lastweekinaws.com/">Bucket Negligence Award</a>, so let’s stay off Corey’s radar. Unfortunately, the CLI dumps policies into escaped JSON strings; and, fortunately jq can deal with that just fine:</p><pre>aws s3api get-bucket-policy --bucket elevenos-production-payloads | jq &quot;.Policy | fromjson&quot;</pre><p>To get the policies for all buckets may take a while, so let’s pipe them to a file (note we have to send the CLI’s annoying error messages to null to get clean output of the policies):</p><pre>{ for bucket in $(aws s3api list-buckets --query &quot;Buckets[].Name&quot; --output text); do aws s3api get-bucket-<br>policy --bucket $bucket 2&gt;/dev/null; done; } &gt; bucket-policies.txt</pre><p>Next we can --slurp them all up with jq and start to see the big picture:</p><pre>jq -s &quot;[ .[].Policy|fromjson|.Statement[] ] | group_by(.Effect) | .[] |  { Effect: .[0].Effect, Principals<br>: [ .[].Principal ] | unique }&quot; bucket-policies.txt</pre><p>Seeing stars?</p><h4>Which of my EC2 Security Groups are being used?</h4><p>This is in response to a suggestion from <a href="https://medium.com/u/b42df2fcb255">A Rick Anderson</a> that got me thinking about cleaning up the detritus of development, and ways to make that easier. While it’s true that following “configuration as code” makes cleaning-up easier (just delete the stack, right?) nonetheless reality prevails. So, true or false, are you using each of your security groups?</p><pre>{ aws ec2 describe-network-interfaces | jq &#39;[.NetworkInterfaces[].Groups[]|.]|map({ (.GroupId|tostring): true }) | add&#39;; aws ec2 desc<br>ribe-security-groups | jq &#39;[.SecurityGroups[].GroupId]|map({ (.|tostring): false })|add&#39;; } | jq -s &#39;[.[1], .[0]]|add|to_entries|[group_by(.value)[]|{ (.[0]<br>.value|if . then &quot;in-use&quot; else &quot;unused&quot; end): [.[].key] }]|add&#39;</pre><h4>Do I have CloudFront Distributions with old TLS Versions?</h4><p>Another security-related one here. The definition of “secure” HTTPS is a moving target as new exploits are discovered and support for new versions of the TLS specification are deployed by AWS. Clicking-though the UI to check each and every origin in your distributions can be, well, a lot of clicks. This one liner will save you some finger strain:</p><pre>aws cloudfront list-distributions \<br> | jq &quot;.DistributionList.Items|.[]|{(.Id):([ .Origins.Items[]|{ (.Id): .CustomOriginConfig.OriginSslProtocols.Items }]|add)}|add&quot;</pre><p>The output is a nice map of distribution IDs to maps of Origin IDs to an array of configured versions:</p><pre>{<br>  &quot;ABC123...&quot;: {<br>    &quot;origin1&quot;: [<br>      &quot;TLSv1.2&quot;<br>    ],<br>    &quot;origin2&quot;: null<br>  },<br>  &quot;CDE456...&quot;: {<br>    &quot;origin3&quot;: [<br>      &quot;TLSv1&quot;,<br>      &quot;TLSv1.1&quot;,<br>      &quot;TLSv1.2&quot;<br>    ]<br>}</pre><h4>Which AWS Regions Support the Services my Solution Needs?</h4><p>Or perhaps asked differently, “Can I deploy my service in region `X`&quot;? This isn’t always an easy thing to figure out from the reference guides and console, but there is bash solution of course! The output of the command below will give you the list of regions that support all of the named services. Note, if you change the list of services, make sure to also change the number in the awk program (bold) to match:</p><pre>for service in &quot;lambda&quot; &quot;stepfunctions&quot; &quot;s3&quot; &quot;comprehendmedical&quot;; do aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/services/$service/regions --query &#39;Parameters[].Value&#39; --output text | tr &#39;[:blank:]&#39; &#39;\n&#39; | grep -v -e ^cn- -e ^us-gov- | sort -r; done | awk &#39;++a[$0]==4{ print $0 }&#39;</pre><p>Example output (comprehendmedical really shortens the list!):</p><pre>us-west-2<br>us-east-2<br>us-east-1<br>eu-west-2<br>eu-west-1<br>ca-central-1<br>ap-southeast-2</pre><h4>Where is the Instance with this Public IP?</h4><p>Somewhere in your sprawling estate of AWS accounts, in some region, there is an instance with an IP. Need to find it <em>now</em> without much fuss or tracing down CIDRs? Update the profile, region lists and &lt;your ip&gt; to suit your needs.</p><pre>for profile in default; \<br>  do for region in us-west-2 us-east-1 eu-west-1; \<br>    do aws ec2 describe-instances \<br>      --region $region \<br>      --profile $profile; \<br>    done; <br>  done <br>| jq -s &quot;[.[].Reservations[].Instances[]|select(.PublicIpAddress != null)|\&quot;ip: \&quot; + .PublicIpAddress + \&quot;, id: \&quot; + .InstanceId]&quot; | grep &quot;&lt;your ip&gt;&quot; </pre><h4>Suggestions?</h4><p>Have suggestions for more interesting questions and/or examples? Put ’em in the comments, below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d54e2eabaf1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1">AWS CLI with jq and Bash</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Patterns for AWS Lambda Fanout]]></title>
            <link>https://medium.com/circuitpeople/patterns-for-aws-lambda-fanout-a7662bb0b2aa?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/a7662bb0b2aa</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[concurrency]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 21:05:10 GMT</pubDate>
            <atom:updated>2018-10-26T20:29:35.296Z</atom:updated>
            <content:encoded><![CDATA[<h4>Getting more done, faster without going back to EC2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2akRLzwZ27vZ-M08tY8bA.png" /></figure><p>A single Lambda execution is limited in terms of RAM, storage, compute and run time. These limits are no barrier for small work units that fit nicely within them, but less so for other cases with large amounts of work to be done. In the latter cases these limits start to get in the way, and that is the motivation for this article — to enumerate some of the key approaches to using Lambda to break out of the box and provide very large amounts of compute in an extremely rapid way.</p><h3><strong>A Single Invocation</strong></h3><p>The default, and simplest, way to use lambda is to connect with an event source such as S3, CloudWatch alarms, DynamoDB, and others. When used in this way, AWS will invoke your Lambda once per event. This mechanism provides for running a short-lived piece of code needing modest resources, and supports both synchronous and asynchronous invocations.</p><p>TheEvent model (aka. “fire and forget”) takes a JSON payload and triggers processing asynchronously. This model doesn’t return a result, but it’s a great fit for tasks like recording a log message or triggering a response to an alarm event. The Event model includes automatic retry when the Lambda produces an error response.</p><pre>var lambda = new AmazonLambdaClient();<br>var result = await lambda.InvokeAsync(new InvokeRequest() {<br>  FunctionName = &quot;my-function&quot;,<br>  InvocationType = <strong>InvocationType.Event</strong>,<br>  Payload = JsonConvert.SerializeObject(new { foo = 1 })<br>});<br>/* response is not available, lambda is running and this process<br>   isn&#39;t blocked. */</pre><p>Alternatively, there is also the RequestResponse model (aka. “I need answer”) which synchronously processes the payload and returns a response once the execution is complete. There is no retry here. This model is used by API Gateway’s Lambda integration (consider that HTTP requires an immediate response) to ensure 1:1 matching of requests and responses.</p><pre>var lambda = new AmazonLambdaClient();<br>var result = await lambda.InvokeAsync(new InvokeRequest() {<br>  FunctionName = &quot;my-function&quot;,<br>  InvocationType = <strong>InvocationType.RequestResponse</strong>,<br>  Payload = JsonConvert.SerializeObject(new { foo = 1 })<br>});<br>/* blocked above until the response is provided */<br>var response = JsonConvert.DeserializeObject&lt;Dictionary&lt;string, object&gt;(result.<strong>Payload</strong>);</pre><p>Work to be done in such Lambdas might involve running a query against S3, Dynamo, ES or RDS, for example, or doing some small calculation.</p><p>To summarize the characteristics of the Single Invocation model:</p><ul><li>It’s simple. Particularly the RequestResponse model which operates much like a remote procedure call. It feels familiar to most developers.</li><li>Both synchronous and asynchronous calling models are available.</li><li>The run time can be no more than 15 minutes (previously only 5 minutes) in duration.</li><li>Processing capacity it limited to 2 ECUs and 3GB of RAM, currently.</li><li>Each function instance may use up to 500MB of temporary storage (ephemeral).</li><li>There is a limit on the maximum number of invocations that can be running at any given time (i.e. the concurrency limit). Ultimately, this limit is the “ceiling” on the compute capacity available under Lambda in your account. It can be increased to absolutely insane levels by simply asking via a Service Limit Increase request in the AWS support console.</li><li>Charges are currently billed on 100ms slices of execution time. If a Lambda isn’t executing, it costs almost nothing. This is the primary reason to care about Lambda at all IMHO — never pay for idle.</li></ul><p>In my previous article, <a href="https://medium.com/circuitpeople/serverless-large-file-downloads-to-s3-a11b4ef4788e">Serverless Large File Downloads to S3</a>, I went into detail on a use case where a single Lambda execution simply can’t get the job done. The culprit in the article was the limited network bandwidth and run time which together restrict the size of content that can be moved. That is one example of “hitting the wall” with Lambda.</p><h3>Getting Further with Lambda</h3><h4><strong>Self-Extending Invocation</strong></h4><p>This is an option to look at when the work unit being processed by a Lambda takes more than 15 minutes, for whatever reason, but the workload is resumable and there is no need for a synchronous result (i.e. using the Event invocation type is already in place, or can be used).</p><p>Resumability means a subsequent execution can pick-up where the previous one left off. This requires little extra code. The Lambda just needs to detect and handle “continuation” state in the payload in addition to handling a normal “start from scratch” invocation payload. Basically, this comes down to testing for the existence of a key holding a cursor or token. An example of this method might be processing a large number of S3 objects where the continuation token would be the key of the last object processed.</p><pre>public async Task&lt;object&gt; MyFunction(JToken input, ILambdaContext context)<br>{<br>  var my_continuation_key = input[&quot;continuation&quot;]?.Value&lt;string&gt;();<br>  // long enough to cleanly exit<br>  var grace = TimeSpan.FromSeconds(10);<br>  while (still_doing_work)<br>  {<br>    /* process a batch... */<br>    if (<strong>context.RemainingTime &lt; grace</strong>) // running out of time...<br>    {<br>      input[&quot;continuation-key&quot;] = new_continuation_key;<br>      var child = new InvokeRequest()<br>      {<br>        FunctionName = context.FunctionName,<br>        InvocationType = InvocationType.Event,<br>        Payload = JsonConvert.SerializeObject(input)<br>      };<br>      var continuation = await new AmazonLambdaClient().InvokeAsync(child);<br>    }<br>  }<br>}</pre><p>Since the original Lambda execution exits at 15 minutes without the work being completed, there isn’t an opportunity for the original caller of the Lambda to receive a result via RequestResponse. Instead, the result will need to be communicated by the final continuation. This might mean storing the result somewhere, or calling an HTTP callback, sending an SNS message, etc.</p><p>Summary of the Self-Extending continuation approach:</p><ul><li>Overcomes the run time limit of a single lambda, and effectively provides an unlimited duration.</li><li>Because continuations are executed with the Event model, there is no option to provide a synchronous result to the original caller.</li><li>The price is comparable to EC2 since the Lambda is always running while there is work to be done. On the plus side, the cost is not incurred while work isn’t being done. Overall, the cost comparison to using EC2 or containers will depend on the duty cycle (percent of time on versus off).</li></ul><h4><strong>Recursive (Self) Invocation</strong></h4><p>When a response is needed by the caller, and more work is on hand than a single invocation can handle, a Lambda can synchronously spawn child invocations to perform concurrent tasks and shorten the overall time. In the special case where the same Lambda is used for both the parent and child invocations, it is self-invocation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*HhkQNj1-dKvH1GKnbRVbnQ.png" /><figcaption>In serverless architectures, <em>Fanout</em> is the term that describes a recursive pattern of invocation where each spawns more than one child. At each layer of recursion the degree of concurrency is multiplied, quickly reaching levels largely unattainable on dedicated hardware.</figcaption></figure><p>The reduction in running time can be impressive when work is well subdivided. The ultimate degree of potential parallelism is determined by the concurrency limit in your account (or set in the Lambda configuration), which can be increased to absurd levels via a support request to AWS. The key to success for this approach is that the processing is <em>associative</em> — changing the order of the processing doesn’t change the outcome. This is an important distinction from the Self-Invoking Continuation, above, where the processing is ordered from start to finish. Or, said more plainly, concurrency.</p><p>Since the caller uses RequestResponse the run time is still limited to 15 minutes, and the initial invocation <strong>must</strong> return a result by then. But, there are options here: If all child invocations have returned and any subsequent processing is completed before the timeout expires, then the final result is normative (i.e. complete). However, a partial result (informative) can also be returned if some work remains to be completed (probably delegated to a continuation). This requires some way to communicate the final status out-of-band, of course.</p><pre>public async Task&lt;object[]&gt; MyFunction(JToken input, ILambdaContext context)<br>{<br>  // your work items in an array<br>  var work = input[&quot;work&quot;].Value&lt;object[]&gt;(); <br>  var itemsPerInvocation = 1000;<br>  // tune to ensure completion within time limit<br>  var batches = work.Select((o, i) =&gt; new { i, o })<br>    .ToLookup(a =&gt; a.i / itemsPerInvocation, a =&gt; a.o)<br>    .Select(g =&gt; g.ToArray());<br>  if (batches.Length &gt; 1) // send it to children<br>  {<br>    var lambda = new AmazonLambdaClient();<br>    var children = batches.Select(b =&gt; new InvokeRequest()<br>      {<br>        FunctionName = context.FunctionName,<br>        InvocationType = <strong>InvocationType.RequestResponse</strong>,<br>        Payload = JsonConvert.SerializeObject(new { work = b})<br>      })<br>      .Select(r =&gt; lambda.InvokeAsync(r))<br>      .ToArray();<br>    // wait for children to <strong>complete</strong><br>    var responses = await Task.WhenAll(children);<br>    // deserialize and combine responses<br>    var strings = responses.Select(r =&gt; new StreamReader(r.Payload).ReadToEnd());<br>    var results = strings.SelectMany(s =&gt; JsonConvert.DeserializeObject&lt;object[]&gt;(s));<br>    return results.ToArray();<br>  }<br>  else<br>  {<br>    /* do the work on a single batch of itemsPerInvocation<br>       items or less */<br>    // return your results<br>    return new object[] { };<br>  }<br>}</pre><p>An example use case for this pattern would be to apply map-reduce to determine the aggregate number of words contained in text files in a large bucket. This is a great fit because the order in which the files are processed has no impact on the result (addition is associative). In this example, the “parent” invocation might perform a ListBucket operation and pass each batch of returned keys as the payload to a child invocation for word counting. As long as each batch is processed and the counts summed within 5 minutes this model would work well. Some tuning of batch sizes might be needed to ensure children return in a timely manner.</p><p>The time required for the Lambda API calls to spawn child processes can be a problem. When the parallelism is high (e.g. for small batch sizes), this time can make a significant dent in the 5 minute limit. A potential answer here is to use grand-children (i.e. having the child invocation invoke multiple “grand child” executions to process sub-batches in parallel, and make more Lambda API calls in parallel — with obvious consequences).</p><p>Summary:</p><ul><li>Can be simple single-layer (children) or multi-layer (grand children) for greater parallelism</li><li>May make use of fixed (hard coded) or dynamic (determined by the data) strategies for spawning child executions</li><li>Retains the 5 minute execution time limit (clock time) but CPU time is multiplied by degree of parallelism</li><li>Provides a synchronous result to the caller</li><li>Maximum speed-up depends on Lambda concurrency and child invocation latency (time per API call)</li><li>Execution blocks (fails) when the concurrency limit is hit</li></ul><h4><strong>Asynchronous Self-Invocation</strong></h4><p>When speed and scale are needed and a synchronous response isn’t, asynchronous self-invocation can churn through huge amounts of work and realize novel value by doing it very quickly.</p><p>The substantial difference from the synchronous strategy, above, is the that the children are invoked using the Event invocation type instead of the RequestResponse model. In the previous strategy the parent invocations must remain running to receive the response from their children, which creates problematic issues when invocations are throttled that limit the effective parallelism. By using the Event invocation model in conjunction with an early exit of the parent, work can be continued and parallelism maximized. The sole responsibility of the parent invocation is to spawn as many child invocations as needed to complete the work.</p><p>An important implication here is that the maximum concurrency limit can be hit very quickly. But unlike with the RequestResponse model, the Event model will cause Lambda to schedule the invocation for when concurrency become available — all with no effective blocking. In this approach both the parent and “child” invocations may use the self-extending invocation pattern safely, if needed.</p><pre>public async Task&lt;object[]&gt; MyFunction(JToken input, ILambdaContext context)<br>{<br>  // your work items in an array<br>  var work = input[&quot;work&quot;].Value&lt;object[]&gt;(); <br>  // tune to ensure completion within time limit<br>  var itemsPerInvocation = 1000;<br>  var batches = work.Select((o, i) =&gt; new { i, o })<br>    .ToLookup(a =&gt; a.i / itemsPerInvocation, a =&gt; a.o);<br>  if (batches.Count &gt; 1) // send it to children<br>  {<br>    var lambda = new AmazonLambdaClient();<br>    var children = batches.Select(g =&gt; new InvokeRequest()<br>      {<br>        FunctionName = context.FunctionName,<br>        InvocationType = <strong>InvocationType.Event</strong>,<br>        Payload = JsonConvert.SerializeObject(input)<br>      })<br>      .Select(r =&gt; lambda.InvokeAsync(r))<br>      .ToArray();<br>    // wait for children to <strong>start</strong><br>    var responses = await Task.WhenAll(children);<br>    // no responses here<br>  }<br>  else<br>  {<br>    /* do the work on a single batch of itemsPerInvocation<br>       items or less */<br>  }</pre><p>An example of recursive self-invocation might be a version of word counting finding that writes the result to an object in an S3 bucket rather than returning it directly to the caller.</p><p>Summary:</p><ul><li>Provides for single-layer or multi-layer recursion (splitting of work can be done less batching occurs in children as well, for example splitting the work by 2, 10 or some other constant that results in smaller and smaller batches as recursion depth increases)</li><li>The division of labor can fixed, and in the code above, or dynamic partitioning</li><li>No synchronous result is available (requires back-channel, probably via an event source like S3, SNS or DynamoDB)</li><li>Scalability depends on Lambda invocation limit, and gracefully handles throttling</li></ul><h3>Patterns to Avoid</h3><p>No list of patterns would be complete without anti-patterns. Or so it seems. I’ve made some blunders along the way, and here are some I would like not to repeat, and think you’ll feel the same way:</p><ul><li>Waiting. If your invocation is just sitting around waiting for something to happen, it’s a bad use of your money. Consider moving to a more event-driven architecture with a service that is an event source to avoid this. AWS Step Functions can be your friend here.</li><li>Cycles. Just don’t. Does the output of a function end up back in the input? Are you *sure*. Common ways of getting here include S3 event handlers that drop objects back in S3, where they get handled, again. Another way is to have a handler on an SQS deal letter queue (DLQ) that re-posts the messages to the queue — chances are this is a terrible idea.</li><li>You probably shouldn’t try combining Recursive (Self) Invocation with Self-Extending invocation. That doesn’t work. If the caller expects a response, you only have 15 minutes to generate it, no matter how much you wish it to be otherwise.</li><li>Latency. Lambda invocations (recursive or otherwise) involve an HTTP API call, and hence suffer from non-trivial latency — particularly across regions. Keep this overhead in mind when choosing work unit size and plurality, because there are only so many API calls that can be made in 15 minutes (it’s lot, but still, be aware of the latency).</li><li>Failure to clean-up. It’s tempting to think that child invocations will be terminated when the parent errors-out, but that’s not the case. Even with the RequestResponse model, child invocations continue to execute even while the parent has long since exited and won’t ever see the response.</li></ul><h3>Summary</h3><p>Even with the recently announced increase from 5 minutes to 15 minutes, there is still a limit on the amount of work a single Lambda invocation can do. You may end up CPU limited, network limited or just plain time limited but if you want to get more done you don’t have to resort to EC2, containers or other forms of servers. Just fan it out, and scale out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7662bb0b2aa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/patterns-for-aws-lambda-fanout-a7662bb0b2aa">Patterns for AWS Lambda Fanout</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Function-per-Resource API Gateways]]></title>
            <link>https://medium.com/circuitpeople/getting-started-with-function-per-resource-api-gateways-69da3c4d48db?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/69da3c4d48db</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[dotnet-core]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Tue, 07 Aug 2018 16:21:44 GMT</pubDate>
            <atom:updated>2018-08-06T23:56:29.298Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been using dotnet Core for much of my development work recently, but I find a couple things about the templates provides by the AWS Tools for it a little frustrating. It’s not the tools themselves — they are low-friction, low ceremony and high quality — but the projects they create for serverless web applications. The crime? They encourage the creation of monolithic functions.</p><p>The default project includes a serverless.template and a single Program.cs file implementing the handler. From here, it’s too easy (and common) to simply add more handlers to the same project and end up with a situation where all the API handlers are in one Lambda.</p><p>Now, people can disagree about having <a href="https://hackernoon.com/aws-lambda-should-you-have-few-monolithic-functions-or-many-single-purposed-functions-8c3872d4338f">few or many functions</a>, but I lean toward the latter in the hope that there is <a href="https://programmingisterrible.com/post/176657481103/repeat-yourself-do-more-than-one-thing-and">value in repeating myself</a>.</p><p>Let’s back up a bit and review why I feel (and yes, these are <em>opinions</em>, slightly different than Yan’s) monolithic functions are harmful:</p><ul><li>They create <strong>high-risk deployments</strong> by putting multiple endpoints and functions at risk for every small change.</li><li>New team members find <strong>increased code complexity and a high learning curve</strong> because the scope of understanding is expanded. It is more difficult to limit the range of understanding to a single thing.</li><li>All team members experience <strong>task inflation</strong> where even simple changes require extensive review to ensure they are, indeed, isolated from other flows encapsulated in the same project. This is made worse by code sharing, below.</li><li>The temptation to create <strong>more internal interconnections and coupling</strong> (i.e. code sharing, aka. “Helpers.csproj” and “Utilities.csproj”) is inevitably realized. Not all code sharing is bad, but when it is bad it’s <em>dreadful</em>.</li><li>With multiple concerns come multiple dependencies and the <strong>hidden complexity</strong> that ensues. If a function supports multiple paths it will need routing. If it has multiple request and responses it will need mapping and serialization libraries. And then an IOC container, because different combinations may be used in production or testing. Each addition obliges developers to more cognitive load and creates opportunities for missed details (now and down the road).</li></ul><p>Please don’t mistake this article being about dotnet Core’s weakness, or API Gateway’s weakness. As you’ll see below, both are extremely capable and support decomposing API implementations into single-purpose functions with ease and grace. And Flask does for Python folks just what ASP.Net Core does for the dotnet folks, and numerous technologies do as well, I’m sure. It’s tempting to continue to do things in familiar ways, and for many single-purpose functions just aren’t familiar.</p><p>If what we want is a function per resource, but the templates don’t offer this option, is there a clean and simple way to build and deploy them with AWS SAM and dotnet Core? I think there is, though it isn’t obvious, and that’s the point of this article/walk-through.</p><p>I’ve tried to make this copy-paste repeatable so you can try it yourself. First, I’ll setup the basic directory and code skeleton using the dotnet and AWS tooling, then I’ll make a few modifications to get to function-per-resource goodness. I’m assuming you already have dotnet Core (preferably version 2.1 or better) and the AWS CLI installed. If you don’t, well, this may not be the right article for you.</p><p>Step 1, install the AWS project templates for dotnet Core:</p><pre>dotnet new -i Amazon.Lambda.Templates</pre><p>The output of that command includes the descriptions of the new templates. I’m going to use just two of them to create three projects:</p><ul><li>A serverless application, which includes a SAM template and a lambda function.</li><li>Two empty Lambda functions (lambda1 and lambda2) that will contain the implementations for two different API endpoints.</li></ul><p>Step 2, create the projects:</p><pre>dotnet new serverless.EmptyServerless -n api</pre><pre>dotnet new lambda.EmptyFunction -n lambda1</pre><pre>dotnet new lambda.EmptyFunction -n lambda2</pre><p>Though it isn’t necessary, for ease of building things I’ll also add a solution file. The main advantage here is that dotnet build and dotnet test will use the solution and build or test, respectively, all of the projects in one command.</p><pre>dotnet new sln -n fpr</pre><pre>dotnet sln fpr.sln add api\src\api\api.csproj lambda1\src\lambda1\lambda1.csproj lambda2\src\lambda2\lambda2.csproj</pre><p>At this point the boilerplate project directory looks like this:</p><pre>fpr.sln<br>api<br>api\src<br>api\test<br>api\src\api<br>api\src\api\api.csproj<br>api\src\api\aws-lambda-tools-defaults.json<br>api\src\api\Function.cs<br>api\src\api\Readme.md<br>api\src\api\serverless.template<br>api\test\api.Tests<br>api\test\api.Tests\api.Tests.csproj<br>api\test\api.Tests\FunctionTest.cs<br>lambda1<br>lambda1\src<br>lambda1\test<br>lambda1\src\lambda1<br>lambda1\src\lambda1\aws-lambda-tools-defaults.json<br>lambda1\src\lambda1\Function.cs<br>lambda1\src\lambda1\lambda1.csproj<br>lambda1\src\lambda1\Readme.md<br>lambda1\test\lambda1.Tests<br>lambda1\test\lambda1.Tests\FunctionTest.cs<br>lambda1\test\lambda1.Tests\lambda1.Tests.csproj<br>lambda2<br>lambda2\src<br>lambda2\test<br>lambda2\src\lambda2<br>lambda2\src\lambda2\aws-lambda-tools-defaults.json<br>lambda2\src\lambda2\Function.cs<br>lambda2\src\lambda2\lambda2.csproj<br>lambda2\src\lambda2\Readme.md<br>lambda2\test\lambda2.Tests<br>lambda2\test\lambda2.Tests\FunctionTest.cs<br>lambda2\test\lambda2.Tests\lambda2.Tests.csproj</pre><p>Each project has it’s own source tree and can be built and tested separately. That’s a good start.</p><p>Speaking of building things, at this point I’ll prepare deployment packages for each of the two Lambdas. Don’t worry too much about what this means or does at this point, just remember that this is different from dotnet build and necessary before deploying the stack. I’ll go into why in the next step, below.</p><pre>pushd lambda1\src\lambda1<br>dotnet lambda package<br>popd</pre><pre>pushd lambda2\src\lambda2<br>dotnet lambda package<br>popd</pre><p>At this point we have two zip files — one for each Lambda — in their respective packaging directories. These zip files contain the full dependencies for each Lambda in the format needed to deploy them. The key here, though, is that these packages only exist <em>locally</em> at this point.</p><p>Step 3 is to get the web application to hook the API Gateway resources and methods directly to these packages. This is where the behavior of the tooling is annoying. It would be <em>excellent </em>if we could simply update serverless.template to point to those projects. And in fact it is almost that easy, but not quite. We need to change the CodeUri property of the functions declared in the SAM template to point to the exact location of zip packages for lambda1 and lambda2:</p><pre>code api\src\api\serverless.template</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*v_gZoSMAWFRumhTVrg-n3Q.png" /><figcaption>Editing the serverless.template file in ./api/src/api to point it to the package output for lambda1 and lambda2. Note the ugly path.</figcaption></figure><p>For convenience, here is what the template ends-up looking like, with two AWS::Serverless::Function resources with their CodeUri properties pointing to the sub-directories where the packages reside:</p><pre>{<br>  &quot;AWSTemplateFormatVersion&quot;: &quot;2010-09-09&quot;,<br>  &quot;Transform&quot;: &quot;AWS::Serverless-2016-10-31&quot;,<br>  &quot;Description&quot;: &quot;An AWS Serverless Application.&quot;,<br>  &quot;Resources&quot;: {<br>    &quot;Get1&quot;: {<br>      &quot;Type&quot;: &quot;AWS::Serverless::Function&quot;,<br>      &quot;Properties&quot;: {<br>        &quot;Handler&quot;: &quot;api::api.Functions::Get&quot;,<br>        &quot;Runtime&quot;: &quot;dotnetcore2.1&quot;,<br>        &quot;CodeUri&quot;: &quot;../../../lambda1/src/lambda1/bin/Release/netcoreapp2.1/lambda1.zip&quot;,<br>        &quot;MemorySize&quot;: 256,<br>        &quot;Timeout&quot;: 30,<br>        &quot;Role&quot;: null,<br>        &quot;Policies&quot;: [<br>          &quot;AWSLambdaBasicExecutionRole&quot;<br>        ],<br>        &quot;Events&quot;: {<br>          &quot;PutResource&quot;: {<br>            &quot;Type&quot;: &quot;Api&quot;,<br>            &quot;Properties&quot;: {<br>              &quot;Path&quot;: &quot;/1&quot;,<br>              &quot;Method&quot;: &quot;GET&quot;<br>            }<br>          }<br>        }<br>      }<br>    },<br>    &quot;Get2&quot;: {<br>      &quot;Type&quot;: &quot;AWS::Serverless::Function&quot;,<br>      &quot;Properties&quot;: {<br>        &quot;Handler&quot;: &quot;lambda2::lambda2.Functions::Get&quot;,<br>        &quot;Runtime&quot;: &quot;dotnetcore2.1&quot;,<br>        &quot;CodeUri&quot;: &quot;../../../lambda2/src/lambda2/bin/Release/netcoreapp2.1/lambda2.zip&quot;,<br>        &quot;MemorySize&quot;: 256,<br>        &quot;Timeout&quot;: 30,<br>        &quot;Role&quot;: null,<br>        &quot;Policies&quot;: [<br>          &quot;AWSLambdaBasicExecutionRole&quot;<br>        ],<br>        &quot;Events&quot;: {<br>          &quot;PutResource&quot;: {<br>            &quot;Type&quot;: &quot;Api&quot;,<br>            &quot;Properties&quot;: {<br>              &quot;Path&quot;: &quot;/2&quot;,<br>              &quot;Method&quot;: &quot;GET&quot;<br>            }<br>          }<br>        }<br>      }<br>    }<br>  },<br>  &quot;Outputs&quot;: {<br>    &quot;ApiURL&quot;: {<br>      &quot;Description&quot;: &quot;API endpoint URL for Prod environment&quot;,<br>      &quot;Value&quot;: {<br>        &quot;Fn::Sub&quot;: &quot;<a href="https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/">https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/</a>&quot;<br>      }<br>    }<br>  }<br>}</pre><p>That’s not awesome. I would prefer to have the tooling just figure out from the handler and runtime that it should package the respective projects and use the output. Something like dotnet lambda deploy fpr.sln would be wonderful.</p><p>Step 4, nonetheless, is to prepare the template for publishing using the AWS CLI:</p><pre>aws cloudformation package --template-file api\src\api\serverless.template --output-template-file after.template --s3-bucket &lt;your_bucket_here&gt;</pre><p>A couple of things happen here:</p><ul><li>The template is processed to find uses of local path in <a href="https://docs.aws.amazon.com/cli/latest/reference/cloudformation/package.html">specific properties</a>.</li><li>It then uploads what it finds at those local paths to the S3 bucket.</li><li>Then it rewrites the template replacing the local paths with the respective S3 URIs for each.</li><li>Finally, it uploads the rewritten template to S3 as well.</li></ul><p>Here is what the output looked like for me:</p><pre>Uploading to e59b3f1f74b6e8d37d03c022a09a483b  203349 / 203349.0  (100.00%)<br>Successfully packaged artifacts and wrote output template to file after.template.<br>Execute the following command to deploy the packaged template<br>aws cloudformation deploy --template-file .\after.template --stack-name &lt;YOUR STACK NAME&gt;</pre><p>And, here are the contents of after.template:</p><pre>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Description: An AWS Serverless Application.<br>Outputs:<br>  ApiURL:<br>    Description: API endpoint URL for Prod environment<br>    Value:<br>      Fn::Sub: <a href="https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/">https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/</a><br>Resources:<br>  Get1:<br>    Properties:<br>      CodeUri: s3://&lt;your_bucket&gt;/ae1cd48289b5be8b1a71a2116180d939<br>      Events:<br>        PutResource:<br>          Properties:<br>            Method: GET<br>            Path: /1<br>          Type: Api<br>      Handler: api::api.Functions::Get<br>      MemorySize: 256<br>      Policies:<br>      - AWSLambdaBasicExecutionRole<br>      Role: null<br>      Runtime: dotnetcore2.1<br>      Timeout: 30<br>    Type: AWS::Serverless::Function<br>  Get2:<br>    Properties:<br>      CodeUri: s3://&lt;your_bucket&gt;/e59b3f1f74b6e8d37d03c022a09a483b<br>      Events:<br>        PutResource:<br>          Properties:<br>            Method: GET<br>            Path: /2<br>          Type: Api<br>      Handler: lambda2::lambda2.Functions::Get<br>      MemorySize: 256<br>      Policies:<br>      - AWSLambdaBasicExecutionRole<br>      Role: null<br>      Runtime: dotnetcore2.1<br>      Timeout: 30<br>    Type: AWS::Serverless::Function<br>Transform: AWS::Serverless-2016-10-31</pre><p>Note that the two Lambda functions reference different S3 objects in their CodeUri properties. When this template is used to create a stack it will include deploying two completely separate functions, each doing just its One Thing:</p><pre>aws cloudformation deploy --template-file .\after.template --stack-name &lt;YOUR STACK NAME&gt;</pre><p>Viola!</p><p>There is some clean-up of the source tree to do here (no need for the Program.cs in the api folder since we aren’t using it), but my purpose is only to show that while the AWS-provided project template encourage monolithic Lambda functions for web applications, there is an easily accessible alternative that leads to the architecturally sustainable pattern of function-per-resource (or method, or whatever is a single concern).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69da3c4d48db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/getting-started-with-function-per-resource-api-gateways-69da3c4d48db">Getting Started with Function-per-Resource API Gateways</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Incredibly, even looking at the outer exception is discouragingly uncommon.]]></title>
            <link>https://medium.com/circuitpeople/incredibly-even-looking-at-the-outer-exception-is-discouragingly-uncommon-3d1f7aa01a8d?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d1f7aa01a8d</guid>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Sun, 05 Aug 2018 21:58:42 GMT</pubDate>
            <atom:updated>2018-08-05T21:58:41.587Z</atom:updated>
            <content:encoded><![CDATA[<p>Incredibly, even looking at the outer exception is discouragingly uncommon. Using the debugger, as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d1f7aa01a8d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/incredibly-even-looking-at-the-outer-exception-is-discouragingly-uncommon-3d1f7aa01a8d">Incredibly, even looking at the outer exception is discouragingly uncommon.</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CloudWatch Logs vs. Kinesis Firehose]]></title>
            <link>https://medium.com/circuitpeople/cloudwatch-logs-vs-kinesis-firehose-71fc3fd54f8c?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/71fc3fd54f8c</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[firehose]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Tue, 12 Jun 2018 22:52:13 GMT</pubDate>
            <atom:updated>2018-11-16T20:13:54.637Z</atom:updated>
            <content:encoded><![CDATA[<h4>No, I’m not putting-out timber fires.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*MJvHmrl-Pgfkueyh.jpg" /><figcaption>[Julie Barichello, <a href="http://www.mywebtimes.com/lists/2018/04/24/09dc27fbe270458bb4d5dcd87d7e8bcb/index.xml?page=2">http://www.mywebtimes.com</a>]</figcaption></figure><p>Logging infrastructure in AWS is awesome, but how do the different options stack-up against each other and what are the appropriate scenarios in which to use each? Let’s see how two popular choices, AWS CloudWatch Logs and Kinesis Data Firehose compare:</p><h3>Setup</h3><p>None of the current AWS offerings allow us to start sending log records without first setting-up some kind of resource. This is reasonable, of course, because AWS needs to have some data structures in place before messages arrive to ensure they are properly handled.</p><h4>CloudWatch Logs</h4><p>CloudWatch requires a log group and log stream to exist prior to sending messages. The group serves to collect streams together and provide a single place to manage settings (retention, monitoring, access control) across them, and a stream is a sequence of log events from the same source.</p><p>Creating a log group requires only a name, but it’s a good idea to employ the optional tagging to keep things organized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JEhVkgttveB4rDqelENnQ.png" /></figure><p>Likewise, creating a new stream is a simple one-liner:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rcLjcObIcVYEBB-OxOh-Aw.png" /></figure><p>And, once created, the description of the stream is concise and straightforward:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vMOVKQdFkndvwPuEMuSfwA.png" /></figure><p>Trying to create a log group or stream using a name that already exists results in a clear error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDxg11qPOhOBw_jR_I-qSw.png" /></figure><p>Finally, removing the group and stream can be accomplished in a single command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gYSrGHQBOJDldlwB0OCNxw.png" /></figure><p>That’s a nice, easy API.</p><h4>Kinesis Firehose</h4><p>A Kinesis Data Firehose delivery stream is designed to take messages at a high velocity (up to 5,000 records per second) and put them into batches as objects in S3. Firehose requires a delivery stream to exist <em>and be active</em> before messages can be sent, and creating one requires a IAM role and an S3 bucket.</p><p>To demonstrate, first comes the bucket (unless you already have one for logs):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kwlEzi2PUFiAvuCQuAq6nw.png" /></figure><p>Next, the IAM role (see the documentation for more detail on <a href="https://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html">how to setup a access for Firehose</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*heYOGlI7B4-4qfFTzoC0UQ.png" /></figure><p>And, to give the role access to the bucket we need to attach an appropriate policy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDnrzmt6lJMnD0Op3jROlw.png" /></figure><p>And, finally, we can create the delivery stream:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_saM27Bd1FqcEdZodeULA.png" /></figure><p>Yeah, that hurt.</p><p>Next note that it takes some time for a new stream to reach ACTIVE state, at which point it can receive messages. A quick check using describe-delivery-stream will let us know when it’s ready:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqTHAda_lcOJeu4pZk4NHw.png" /></figure><p>And, when we’re done with it we can remove the delivery stream with a one-liner as well (but don’t forget to remove the policy, role and bucket, too):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFrKoD17nb2uMuTPk6vFfg.png" /></figure><p>Setting-up and cleaning-up Firehose takes a lot more work than for CloudWatch Logs.</p><h3>Sending Messages</h3><p>For example purposes, let’s look at simply sending the tail of some local log event file to CloudWatch. A concrete example of this might be something simple like pushing PiHole events from a Raspberry Pi so you can be alerted when blacklisted domains are being requested inside your network (a sign of malware, perhaps).</p><h4>CloudWatch Logs</h4><p>When sending messages to CloudWatch Logs you must know if it is the first message, or a subsequent message. This is important because with the first message we must <em>not</em> provide a sequence number, but we are required to provide a sequence number for all subsequent messages.</p><p>In addition, we must also provide a timestamp for each message and those timestamps must be in increasing order over the messages included in the command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hrS8i8F3XB2lGeNpOnwUMg.png" /></figure><p>The sequence number returned must be provided when sending the next message(s):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*clYJj3W-GmTAs68HSccH8g.png" /></figure><p>Sending the tail lines requires keeping track of the sequence token between calls. One way to do that is in an environment variable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHH7hy31bHhV6mpipQqCTQ.png" /></figure><p>Painful, and brittle. We need to sync the system clock, and set the sequence token via describe-log-stream when the system reboots (etc.) or messages will fail to send. This seems like extra work since it’s almost certain that the log messages contain a date already.</p><h4>Kinesis Firehose</h4><p>The command to send a message is very simple and only uses one parameter: the name of the delivery stream:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8UjsD71Bh3VL0S9HpVimg.png" /></figure><p>To tail the logs to Firehose is simpler than for CloudWatch logs since we don’t need to provide a timestamp or a sequence token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKFucBJP7DCTHrAgPp2E4w.png" /></figure><p>Easy, and reliable.</p><h3>Message Pipeline</h3><p>Getting messages <em>into</em> a logging system is necessary but not sufficient. Getting them out is just as important. We need to use them as event triggers, via search or archival/audit (likely all three).</p><h4>CloudWatch Logs</h4><p>Search over log groups and individual streams is provided in the AWS Console, with no setup needed. The search is basic (keyword and date/time range only) and sometimes quite slow, but it’s easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*8P80xEugWZiCaL-DGR_HUA.png" /><figcaption>Simple search is provided in the console for CloudWatch Logs.</figcaption></figure><p>You’ll also find the console provides ways to create custom metrics and send filtered log stream events to Lambda for open-ended processing in small batches. From an operations perspective, the retention period can be configured from “forever” to as short as a day for a log group.</p><h4>Kinesis Firehose</h4><p>Search over single log files is provided in the S3 Console under the “S3 Select” moniker, but searching over multiple files isn’t. S3 Select is very capable, being based on a SQL-like syntax.</p><p>As for event sourcing, when Firehose creates new objects in S3 events can invoke Lambda for efficient, open-ended large batch processing.</p><p>For high volumes with long retention and deep analysis, Firehose supports RedShift as a destination. RedShift is a sophisticated, scalable and cost-effective analysis solution. Firehose and CloudWatch logs are also commonly used with ElasticSearch for an easy to use analysis option, but with limited scale and comparatively high cost.</p><p>In S3, the log events are stored cheaply, and support random access by time (the key prefix includes the date and hour) and are subject to S3’s powerful data retention policies (send to Glacier, expire, etc.).</p><h3>Pricing</h3><p>Economics matter — but only if you grow out of the free tier. Both CloudWatch and Firehose charge on a metered, pay only for what you you basis so they are genuinely serverless in that regard.</p><h4>CloudWatch Logs</h4><ul><li>$0.50 per GB ingested**</li><li>$0.03 per GB archived per month***</li><li>$1.00 per-million-custom-events-generated*****</li></ul><p>** There is no Data Transfer IN charge for any of CloudWatch.<br>*** Data archived by CloudWatch Logs includes 26 bytes of metadata per log event and is compressed using gzip level 6</p><h4>Kinesis Firehose</h4><ul><li>$0.029 per GB, Data Ingested, First 500 TB / month</li><li>$0.018 per GB, Data Format Conversion (optional)</li><li>$0.023 per GB, S3 Standard Storage</li></ul><h3>Conclusion(s)</h3><p>For quick setup and simple searching of messages, CloudWatch Logs is a winner. The somewhat tedious and sprawling dependencies backing a Firehose means there is more to learn (IAM) and maintain.</p><p>From an operations point of view, sending messages with Firehose is as simple and reliable as can be. The limits on Firehose mean each account may have only 50 delivery streams (more by request) and that each delivery stream can accept up to 5,000 records per second. Other limits on the size of individual records and number of records per batch are important, but they are quite high. And, at high volumes the price differences really add up. For example, with 1TB of logs per month with 90 day retention:</p><ul><li>CloudWatch Logs: 1,024 x $0.50 + 3 x 1,024 x $0.03 = $604.16/month</li><li>Firehose: 1,024 x $0.029 + 3 x 1,024 x $0.023 = $100.35/month</li></ul><p>Winner? I prefer Firehose. Get past the setup and enjoy the cost-effective, reliable and never-worry-about-scaling option. Unless you like it easy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71fc3fd54f8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/cloudwatch-logs-vs-kinesis-firehose-71fc3fd54f8c">CloudWatch Logs vs. Kinesis Firehose</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with dotnet Core and AWS Aurora]]></title>
            <link>https://medium.com/circuitpeople/getting-started-with-dotnet-core-and-aws-aurora-df953533e47a?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/df953533e47a</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[net-core]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Fri, 08 Jun 2018 20:01:54 GMT</pubDate>
            <atom:updated>2018-06-08T20:05:52.241Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s *extremely* interesting to me that <a href="https://aws.amazon.com/blogs/aws/in-the-works-amazon-aurora-serverless/">AWS is working on a serverless implementation of Aurora</a> for on-demand (no reserved capacity) workflows. SQL hasn’t had a place in my serverless architectures in recent years, as I’m allergic to running idle DB instances and the operational complexity of managing cluster on/off/scaling myself. I’m very much looking forward to getting back to relational storage when that product is available.</p><p>Since it’s been a while since I’ve worked in such a space, I recently made a quick run-through of building and running a dotnet core application with EntityFrameworkCore for modeling and using an AWS Aurora cluster as the storage engine. Why? Well, this stack requires code from three (primarily, more actually) vendors to work together, and call me skeptical but I tend to test this kind of thing. Here is what I found.</p><h4>Setting-up an Aurora Cluster</h4><p>Currently AWS Aurora is setup as any other RDS database, so the process involved choosing an engine, instance types, roles and other options. Since I’m just running-though a quick test here I basically just clicked-through the options for a new cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*12TFb19q0mArCJBii5DFJQ.png" /><figcaption>Choose Aurora as the engine. I’m not sure why I have to choose the kind of compatibility (wouldn’t “all of the above” be nice?), but MySQL 5.6 works for me this time.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vh2OZujd0jKxzYHLdWMgsg.png" /><figcaption>Yep, pick your rate of wallet-drain, err, I mean select your instance type. Smaller is better unless it isn’t.</figcaption></figure><p>As with any other (current) RDS option, the instances supporting your database are going to be running 24/7 unless you take responsibility for shutting them down, and restoring from backups when you need them again. The cost may be small-ish, but since separating storage in microservices is a thing, the small costs become large costs.</p><p>Aurora Serverless will, hopefully, mean this step in the process goes away and AWS takes the responsibility of scaling-up and down (to zero) my DB when I’m not using it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s65r-dZwiSBWz9Z2j_Osjw.png" /><figcaption>More settings, yes you should use RDS in a VPC and with a well-crafted security group and ACLs. And a strong master password that you don’t hard-code into your source code control system (or use anywhere, except maybe in disaster recovery). Sorry, that’s a sore point.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LK7V1FClceS3752ZYTn6aw.png" /><figcaption>Encryption is awesome, so says GDPR and NIST. Use it, please. And retain backups for perhaps longer than a day — is that a reasonable default, AWS? Sorry, again, I digress.</figcaption></figure><p>Once through the wizard, you’ll find yourself with a cluster with a primary endpoint (DNS name) for access. Since the DB instances are in a VPC to make life simple we’ll use a bastion host for building the test app.</p><h4>Creating a Bastion Instance</h4><p>To make this simple let’s build the app on an <a href="https://aws.amazon.com/about-aws/whats-new/2018/03/announcing--net-core-ami-for-amazon-ec2/">AWS Linux 2 with dotnet core instance</a> in the same VPC as the cluster, and with the cluster’s security group modified to allow traffic from the instances private IP. Using Windows bash (aka. Ubuntu on Windows), start by grabbing the current list of amazon AMIs and piping it into jq to find the one for .Net Core:</p><pre>$ <strong>export AMI_ID=$(aws ec2 describe-images --owners amazon | jq -r &quot;.Images[] | { id: .ImageId, desc: .Description } | select(.desc?) | select(.desc | contains(\&quot;Amazon Linux 2\&quot;)) | select(.desc | contains(\&quot;.NET Core\&quot;)) | .id&quot;)</strong><br>$ <strong>echo $AMI_ID</strong><br>ami-950c6ced</pre><p>Given the AMI, we’ll spin-up an micro instance. For that I need a key pair, and since this is a throw-away project I’m just going to create one here:</p><pre>$ <strong>aws ec2 create-key-pair --key-name aurora-test-keypair &gt; aurora-test-keypair.pem</strong></pre><p>When creating instances, I like to test the command first using the --dry-run option. Note that you’ll need to grab the subnet ID from the cluster created above:</p><pre>$ <strong>aws ec2 run-instances --instance-type t2.micro --image-id $AMI_ID --subnet-id &lt;your_subnet_id&gt; --key-name aurora-test-keypair --count 1 --dry-run</strong></pre><p>It looks like the command is well-formed and what I want, so I create the instance for real and pipe the output to a file:</p><pre>$ <strong>aws ec2 run-instances --instance-type t2.micro --image-id $AMI_ID --region us-east-1 --subnet-id &lt;your_subnet_id&gt; --key-name aurora-test-keypair --count 1 &gt; instance.json</strong></pre><p>To SSH into the instance I’ll need the public IP address (the DNS name would work, too):</p><pre>$ <strong>export INSTANCE_ID=$(jq -r .Instances[].InstanceId instance.json)<br></strong>...~wait a bit~...<br>$ <strong>export </strong><strong>INSTANCE_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --output text --query &#39;Reservations[*].Instances[*].PublicIpAddress&#39;)</strong></pre><p>And, finally, let’s get onto the instance:</p><pre>$ <strong>ssh -i aurora-test-keypair.pem ec2-user@$INSTANCE_IP</strong></pre><h4>Creating the dotnet Core Application</h4><p>Now on the some coding. Step one for any dotnet core app is to initialize the project:</p><pre>[ec2-user@my-instance]$ <strong>dotnet new console -o aurora-demo</strong><br>The template &quot;Console Application&quot; was created successfully.</pre><pre>Processing post-creation actions...<br>Running &#39;dotnet restore&#39; on aurora-demo/aurora-demo.csproj...<br>  Restoring packages for /home/ec2-user/aurora-demo/aurora-demo.csproj...<br>  Generating MSBuild file /home/ec2-user/aurora-demo/obj/aurora-demo.csproj.nuget.g.props.<br>  Generating MSBuild file /home/ec2-user/aurora-demo/obj/aurora-demo.csproj.nuget.g.targets.<br>  Restore completed in 199.49 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.</pre><pre>Restore succeeded.</pre><pre>[ec2-user@my-instance]$ <strong>cd aurora-demo/</strong></pre><p>Next, add the needed references for EntityFrameworkCore and MySql. Even though we’re not going to be doing any “design time” modeling, the project still needs a reference to Microsoft.EntityFrameworkCore.Design to allow us to use the CLI extensions (more on that below):</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet add package Microsoft.EntityFrameworkCore --version=2.0.3</strong><br>  Writing /tmp/tmpOirB8v.tmp<br>info : Adding PackageReference for package &#39;Microsoft.EntityFrameworkCore&#39; into project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>log  : Restoring packages for /home/ec2-user/aurora-demo/aurora-demo.csproj...<br>info : Package &#39;Microsoft.EntityFrameworkCore&#39; is compatible with all the specified frameworks in project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>info : PackageReference for package &#39;Microsoft.EntityFrameworkCore&#39; version &#39;2.0.3&#39; added to file &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>[ec2-user@my-instance aurora-demo]$ <strong>dotnet add package Microsoft.EntityFrameworkCore.Design --version=2.0.3</strong><br>  Writing /tmp/tmpLwRNxf.tmp<br>info : Adding PackageReference for package &#39;Microsoft.EntityFrameworkCore.Design&#39; into project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>log  : Restoring packages for /home/ec2-user/aurora-demo/aurora-demo.csproj...<br>info : Package &#39;Microsoft.EntityFrameworkCore.Design&#39; is compatible with all the specified frameworks in project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>info : PackageReference for package &#39;Microsoft.EntityFrameworkCore.Design&#39; version &#39;2.0.3&#39; added to file &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>[ec2-user@my-instance aurora-demo]$ <strong>dotnet add package MySql.Data --version 8.0.11</strong><br>  Writing /tmp/tmpj1hhEQ.tmp<br>info : Adding PackageReference for package &#39;MySql.Data&#39; into project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>log  : Restoring packages for /home/ec2-user/aurora-demo/aurora-demo.csproj...<br>info : Package &#39;MySql.Data&#39; is compatible with all the specified frameworks in project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>info : PackageReference for package &#39;MySql.Data&#39; version &#39;8.0.11&#39; added to file &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>[ec2-user@my-instance aurora-demo]$ <strong>dotnet add package MySql.Data.EntityFrameworkCore --version 8.0.11</strong><br>  Writing /tmp/tmpJpOeCZ.tmp<br>info : Adding PackageReference for package &#39;MySql.Data.EntityFrameworkCore&#39; into project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>log  : Restoring packages for /home/ec2-user/aurora-demo/aurora-demo.csproj...<br>info : Package &#39;MySql.Data.EntityFrameworkCore&#39; is compatible with all the specified frameworks in project &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.<br>info : PackageReference for package &#39;MySql.Data.EntityFrameworkCore&#39; version &#39;8.0.11&#39; added to file &#39;/home/ec2-user/aurora-demo/aurora-demo.csproj&#39;.</pre><p>At this point project file should look like this:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>more aurora-demo.csproj </strong><br>&lt;Project Sdk=&quot;Microsoft.NET.Sdk&quot;&gt;</pre><pre>  &lt;PropertyGroup&gt;<br>    &lt;OutputType&gt;Exe&lt;/OutputType&gt;<br>    &lt;TargetFramework&gt;netcoreapp2.0&lt;/TargetFramework&gt;<br>  &lt;/PropertyGroup&gt;</pre><pre>  &lt;ItemGroup&gt;<br>    &lt;PackageReference Include=&quot;Microsoft.EntityFrameworkCore&quot; Version=&quot;2.0.3&quot; /&gt;<br>    &lt;PackageReference Include=&quot;Microsoft.EntityFrameworkCore.Design&quot; Version=&quot;2.0.3&quot; /&gt;<br>    &lt;PackageReference Include=&quot;MySql.Data&quot; Version=&quot;8.0.11&quot; /&gt;<br>    &lt;PackageReference Include=&quot;MySql.Data.EntityFrameworkCore&quot; Version=&quot;8.0.11&quot; /&gt;<br>  &lt;/ItemGroup&gt;</pre><pre>&lt;/Project&gt;</pre><p>EntityFrameworkCore provides some nice extensions to the dotnetcommand that may be enabled by hand-editing the project file. I used nanoto make those changes:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>nano aurora-demo.csproj</strong></pre><pre>[ec2-user@my-instance aurora-demo]$ <strong>more aurora-demo.csproj </strong><br>&lt;Project Sdk=&quot;Microsoft.NET.Sdk&quot;&gt;</pre><pre>  &lt;PropertyGroup&gt;<br>    &lt;OutputType&gt;Exe&lt;/OutputType&gt;<br>    &lt;TargetFramework&gt;netcoreapp2.0&lt;/TargetFramework&gt;<br>  &lt;/PropertyGroup&gt;</pre><pre>  &lt;ItemGroup&gt;<br>    &lt;PackageReference Include=&quot;Microsoft.EntityFrameworkCore&quot; Version=&quot;2.0.3&quot; /&gt;<br>    &lt;PackageReference Include=&quot;Microsoft.EntityFrameworkCore.Design&quot; Version=&quot;2.0.3&quot; /&gt;<br>    &lt;PackageReference Include=&quot;MySql.Data&quot; Version=&quot;8.0.11&quot; /&gt;<br>    &lt;PackageReference Include=&quot;MySql.Data.EntityFrameworkCore&quot; Version=&quot;8.0.11&quot; /&gt;<br>  &lt;/ItemGroup&gt;</pre><pre>  &lt;ItemGroup&gt;<br>    &lt;DotNetCliToolReference Include=&quot;Microsoft.EntityFrameworkCore.Tools&quot; Version=&quot;2.0.3&quot; /&gt;<br>    &lt;DotNetCliToolReference Include=&quot;Microsoft.EntityFrameworkCore.Tools.DotNet&quot; Version=&quot;2.0.0&quot; /&gt;<br>  &lt;/ItemGroup&gt;</pre><pre>&lt;/Project&gt;</pre><p>The bit I added to enable EF tooling is the ItemGroup section containing the two DotNetCliToolReference entries. This tells dotnet that I want to use extra tooling inside this project, and that tooling will be managed like a package reference (i.e. dotnet restore). Once dotnet Core 2.1 is supported this step will no longer be necessary since it bundles these tools into the distribution.</p><p>The dotnet new command, above, also created a basic source code file for a console application. I just added a little code to test that the database is actually being used. Again, a little bit more time in nano:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>nano Program.cs </strong><br>[ec2-user@my-instance aurora-demo]$ <strong>more Program.cs </strong><br>using System;<br>using System.Collections.Generic;<br>using System.Linq;<br>using System.Threading.Tasks;<br>using Microsoft.EntityFrameworkCore;</pre><pre>namespace aurora_demo<br>{<br>   class Program<br>   {<br>      static void Main(string[] args)<br>      {<br>         Console.WriteLine(&quot;Hello World!&quot;);<br>         using (var db = new FooContext())<br>         {<br>            var foo = db.Foos.FirstOrDefault() ?? db.Foos.Add(new Foo()).Entity;<br>            Console.WriteLine(foo?.Bar);<br>            db.SaveChanges();<br>         }<br>      }<br>   }</pre><pre>   public class Foo<br>   {<br>      public int Id { get; set; }<br>      public string Bar { get; set; } = &quot;Baz&quot;;<br>   }</pre><pre>   internal class FooContext : DbContext<br>   {<br>      protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)<br>      {<br>         optionsBuilder.UseMySQL(System.Environment.GetEnvironmentVariable(&quot;FOO_CONNECTION_STRING&quot;));<br>         base.OnConfiguring(optionsBuilder);<br>      }<br>      public DbSet&lt;Foo&gt; Foos { get; set; }<br>   }<br>}</pre><p>Note that I’m pulling the connection string from an environment variable, which I’ll set later (a better way is to use Parameter Store). The code here simply checks that a Foo is in the database, and if not creates one, then returns the value of the Foo&#39;s Bar property.</p><p>Source code changes complete, let’s see how we do pulling-down the referenced packages:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet restore</strong><br>  Restoring packages for /home/ec2-user/aurora-demo/aurora-demo.csproj...<br>  Restore completed in 367.65 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.<br>  Restore completed in 12.26 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.<br>  Restore completed in 5.03 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.</pre><p>So far, so good. But does it build?</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet build</strong><br>Microsoft (R) Build Engine version 15.7.177.53362 for .NET Core</pre><pre>Copyright (C) Microsoft Corporation. All rights reserved.<br></pre><pre>  Restore completed in 50.87 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.<br>  Restore completed in 8.9 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.<br>  Restore completed in 6.07 ms for /home/ec2-user/aurora-demo/aurora-demo.csproj.<br>  aurora-demo -&gt; /home/ec2-user/aurora-demo/bin/Debug/netcoreapp2.0/aurora-demo.dll</pre><pre>Build succeeded.<br>    0 Warning(s)<br>    0 Error(s)</pre><pre>Time Elapsed 00:00:04.09</pre><p>Indeed! Now, on to the interesting bit. As well as testing basic run-time SELECT and INSERT commands I want to see if the tooling works to setup and update the database schema. EF calls this support “migrations”. My first basic test is to have the tooling create the DB for me. But first I need to setup the connection string environment variable.</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>export FOO_CONNECTION_STRING=&quot;server=&lt;your_aurora_cluster_dns_name&gt;;port=3306;database=foo;uid=&lt;your_user_name&gt;;password=&lt;your_password&gt;;SSLMode=None&quot; </strong></pre><p>Specifying the connection must be done before attempting to update the database, obviously. Note the use of SSLMode=None -- that&#39;s unfortunate, but current versions of the Oracle MySql entity framework library defaults to SSL required and that isn&#39;t supported on my cluster.</p><p>As I mentioned before, it would be safer and better to use AWS EC2 Parameter Store (or Secrets Manager) for the connection string, but I’m just keeping things simple for this demo.</p><p>The EF tooling adds the ef sub-command to dotnet. Two of the sub-sub-commands it adds are migrations and database which are used to generate differential change sets (i.e. code) based on snapshots of the DbContext instances in the code, and apply them to the database, respectively.</p><p>To create the initial version of the database we need a schema, and that’s what migrations handle. Creating a migration creates a snapshot of the current state of the code for the data context, and compares that to the previous state (which is empty for the initial migration). To take this first snapshot, I used the add sub-sub-sub-command.</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet ef migrations add Initial</strong><br>Done. To undo this action, use &#39;ef migrations remove&#39;</pre><p>Note that last line of the message — if we want to make more changes to the information model (DbContext) you can simply remove the previous migration and start again. But, we&#39;re good here so let&#39;s just look at the project directory structure:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>ls -R</strong><br>.:<br>aurora-demo.csproj  bin  Migrations  obj  Program.cs</pre><pre>./bin:<br>Debug</pre><pre>./bin/Debug:<br>netcoreapp2.0</pre><pre>./bin/Debug/netcoreapp2.0:<br>aurora-demo.deps.json  aurora-demo.dll  aurora-demo.pdb  aurora-demo.runtimeconfig.dev.json  aurora-demo.runtimeconfig.json</pre><pre>./Migrations:<br>20180527012338_Initial.cs  20180527012338_Initial.Designer.cs  FooContextModelSnapshot.cs</pre><pre>./obj:<br>aurora-demo.csproj.EntityFrameworkCore.targets  aurora-demo.csproj.nuget.cache  aurora-demo.csproj.nuget.g.props  aurora-demo.csproj.nuget.g.targets  Debug  project.assets.json</pre><pre>./obj/Debug:<br>netcoreapp2.0</pre><pre>./obj/Debug/netcoreapp2.0:<br>aurora-demo.AssemblyInfo.cs           aurora-demo.csprojAssemblyReference.cache   aurora-demo.csproj.FileListAbsolute.txt  aurora-demo.pdb<br>aurora-demo.AssemblyInfoInputs.cache  aurora-demo.csproj.CoreCompileInputs.cache  aurora-demo.dll</pre><p>The Migrations folder contains the code-generated migration code -- code generated from code, is that a good thing? Yes, yes it is. Now that we have the migration ready, can we use Microsoft&#39;s code extended with Oracle&#39;s code to talk to Amazon&#39;s code?</p><p>Let’s try:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet ef database update</strong></pre><pre>Applying migration &#39;20180527012338_Initial&#39;.<br>Done.</pre><p>Nice. But does it run?</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet run</strong><br>Hello World!</pre><pre>Unhandled Exception: MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---&gt; System.IO.IOException: Unable to read data from the transport connection: Connection timed out. ---&gt; System.Net.Sockets.SocketException: Connection timed out<br>   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)<br>   --- End of inner exception stack trace ---<br>   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)<br>   at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)<br>   at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)<br>   at MySql.Data.MySqlClient.MySqlStream.LoadPacket()<br>   --- End of inner exception stack trace ---<br>   at MySql.Data.MySqlClient.MySqlStream.LoadPacket()<br>   at MySql.Data.MySqlClient.MySqlStream.ReadPacket()<br>   at MySql.Data.MySqlClient.NativeDriver.Open()<br>   at MySql.Data.MySqlClient.Driver.Open()<br>   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)<br>   at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()<br>   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()<br>   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()<br>   at MySql.Data.MySqlClient.MySqlPool.GetConnection()<br>   at MySql.Data.MySqlClient.MySqlConnection.Open()<br>   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)<br>   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(Boolean buffer)<br>   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()<br>   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean&amp; found)<br>   at lambda_method(Closure , QueryContext )<br>   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.&lt;&gt;c__DisplayClass17_1`1.&lt;CompileQueryCore&gt;b__0(QueryContext qc)<br>   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)<br>   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)<br>   at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable`1 source)<br>   at aurora_demo.Program.Main(String[] args) in /home/ec2-user/aurora-demo/Program.cs:line 16</pre><p>Oops. Not to worry, my cluster is a little <em>special</em> and can take some time to respond to the initial request. So, let’s just try again now that it’s had a chance to warm up:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet run</strong><br>Hello World!<br>Baz</pre><p>Bingo! But did that response actually come from the database? Let’s see what MySql says:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>mysql -h &lt;your_aurora_cluster_dns_name&gt; -u &lt;your_user_name&gt; -p -e &quot;SELECT * FROM foo.Foos;&quot;</strong><br>Enter password: <br>+----+------+<br>| Id | Bar  |<br>+----+------+<br>|  1 | Baz  |<br>+----+------+</pre><p>Yep, that’s the right data in the table. Let’s use MySql to change the information directly and make sure the code responds and reads the new value:</p><pre>[ec2-user@my-instance aurora-demo]$ <strong>mysql -h &lt;your_aurora_cluster_dns_name&gt; -u &lt;your_user_name&gt; -p -e &quot;UPDATE foo.Foos SET Bar=&#39;Bingo&#39; WHERE Id = 1;&quot;</strong><br>Enter password:</pre><pre>[ec2-user@my-instance aurora-demo]$ <strong>mysql -h &lt;your_aurora_cluster_dns_name&gt; -u &lt;your_user_name&gt; -p -e &quot;SELECT * FROM foo.Foos;&quot;</strong><br>Enter password: <br>+----+-------+<br>| Id | Bar   |<br>+----+-------+<br>|  1 | Bingo |<br>+----+-------+</pre><pre>[ec2-user@my-instance aurora-demo]$ <strong>dotnet run</strong><br>Hello World!<br>Bingo</pre><p>Fun. It’s pretty amazing that this works, and a testament to the strength and maturity of these technologies. While there are almost certainly issues I’ve yet to discover, taken with the <a href="https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581">high performance of dotnet core Lambdas</a>, it’s looking like a production stack to me.</p><p>I’m more excited for Aurora Serverless than ever…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df953533e47a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/getting-started-with-dotnet-core-and-aws-aurora-df953533e47a">Getting Started with dotnet Core and AWS Aurora</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless, large file downloads to S3]]></title>
            <link>https://medium.com/circuitpeople/serverless-large-file-downloads-to-s3-a11b4ef4788e?source=rss----ab6443916890---4</link>
            <guid isPermaLink="false">https://medium.com/p/a11b4ef4788e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[step-functions]]></category>
            <dc:creator><![CDATA[Lee Harding]]></dc:creator>
            <pubDate>Fri, 02 Feb 2018 04:34:00 GMT</pubDate>
            <atom:updated>2018-02-02T04:34:00.135Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using AWS Step Functions and Lambda for Fanout</h4><p>I have a love for FaaS, and in particular <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> for breaking so much ground in this space. Many of the most valuable uses I’ve found for Lambda involve cost and performance as core requirements — that is, if the service can be 10x faster or cheaper it will provide disruptive benefits to the customer.</p><p>Fanout is a key mechanism for achieving that kind of cost-efficient performance with Lambda. Fanout is a category of patterns for spreading work among multiple Function invocations to get more done sooner. This is, of course, horizontal scaling (also known as “scaling out”) and works by using many resources to side-step limitations associated with a single resource. Specifically, this might mean getting more CPU cycles in less time, more bytes over the network in less time, more memory, etc.</p><p>An example I like to use here is moving a large file into S3, where there will be a limit on the bandwidth available to the Function *and* a limit on the time the function can run (5 minutes). I’ve done some experiments to demonstrate the effective size of file that can be moved through a Lambda in this way. The image below shows the result of a recent one where a Step Function state machine is used to measure the time to download increasingly large files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*CCDhgYjz2Uxt_PZUyL5arw.png" /><figcaption>Attempting to download a set of increasingly large files. Timings shown in the table below.</figcaption></figure><p>The bottom line here is that files larger than a several GB won’t reliably download in a single Lambda invocation. The effective bandwidth over this range of files sizes varied from 400 to 700 million bits per second. Good, but not enough for moving some interesting things (e.g. the NSRL hashsets, Videos, ML training sets, etc.).</p><pre>Download  MB      ms<br>10MB      10      221<br>100MB     100     1873<br>1GB       1024    16510<br>10GB      10240   190015<br>100GB     102400  failed</pre><p>Note that AWS will very likely improve these numbers — they have a great track record of continuously delivering on such things. Nonetheless, there will always be a limit, and that limit is small enough now to cause problems. Lambda executions can only run for 5 minutes (300,000ms) so extrapolating the data above indicates that downloading anything above about 15GB will consistently fail. Extrapolating further, it looks like the Lambda execution time limit would need to be increased to over 30 minutes for a 100GB file to have a chance of downloading in a single execution.</p><p>At this point we could throw up our hands and go back to long-running transfers on EC2 or an ECS Container, but that would be silly. Fanout is the obvious answer, because:</p><ul><li>We’re moving the file from a website that supports <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests">HTTP Range requests</a> (i.e. we can request a specific sub-section of the file rather than the entire thing).</li><li>S3 supports <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html">Multi-part Uploads</a> (i.e. we can upload different sections of the file into parts, and combine them once completed)</li><li>Lambda Function executions <a href="https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-code.html">run as isolated environments</a> with their own CPU and network capabilities</li></ul><p>By using multiple executions we can download different ranges of the source file in parallel, with each creating a “part” in S3, and then combine the parts once all ranges are complete. To demonstrate the idea, consider this simple prototype with <a href="https://aws.amazon.com/step-functions/">AWS StepFunctions</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*Ikn-IQuLTLsLC652h1uGDg.png" /><figcaption>Simple prototype state machine for downloading files from 0–20GB in size.</figcaption></figure><p>If you aren’t familiar with Step Functions (you might want to be, it’s an excellent tool to have in your kit), the important thing to know here is that each node in the diagram is either a link to a Lambda function to be run (aka. a <a href="https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-task-state.html">Task</a> state), or a flow-control node such as a <a href="https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-choice-state.html">Choice</a>, <a href="https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-pass-state.html">Pass</a> or Parallel state. Choice states allow control to be passed to one of many subsequent nodes based on conditions on the output of the preceding node. Pass states allow simple transformations to be applied to the input before passing it to the next node (without having to do so in a Lambda).</p><p>The core of this state machine is the <a href="https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-parallel-state.html">Parallel</a> state (represented by the dashed border region), which provides concurrency through: Executing its child state machines (aka. branches) asynchronously; waiting for them to complete, and; proceeding to the following node. The output of a Parallel state is an array containing the output of the last node in each child branch.</p><p>In the diagram above the left-most branch contains a single Task that downloads the first part of the file (the other two nodes are Pass states that exist only to format input or output). The other branches contain conditional logic based on the size of the file:</p><ul><li>If the file is larger than the minimum needed by the part, download the appropriate 1/5th of the file. For example the second branch will download and create a part only if the file is larger than 5MB, the third 10MB, etc.</li><li>But if the file is less than 5MB ,(or 10, 15, etc. for the other branches) the download is skipped.</li></ul><p>As you can see, this idea can be scaled-out to allow the download of very large files and with broad concurrency. What are some of the details here?</p><p>The first step is to determine if the source URL supports Ranges would normally be to make an OPTIONS request. AWS S3 endpoints support Ranges but because it’s used for CORS it doesn’t work for simple queries like ours (basically it requires a couple extra headers). So we instead make this check using a HEAD request, which achieves the same result:</p><pre>var request = WebRequest.CreateHttp(url);<br>request.Method = &quot;HEAD&quot;;<br>using (var response = await request.GetResponseAsync())<br>{<br>  var supports_ranges = <br>    response.Headers.AllKeys.Contains(&quot;Accept-Ranges&quot;)<br>    &amp;&amp; response.Headers[&quot;Accept-Ranges&quot;].Contains(&quot;bytes&quot;);<br>  var content_length =<br>    long.Parse(response.Headers[HttpResponseHeader.ContentLength]);<br>  ...<br>}</pre><p>Another HTTP-related detail is how to make a request for a subset of content once we know supports_ranges is true. Not all servers/domains will support ranges. If they don’t, asking for a range may (or may not depending on the server software) cause an error response. In some cases, the range request will simply be ignored and the entire content will be returned.</p><pre>var start = ...index of first byte to be returned<br>var end = ...index of last byte to be returned, _inclusive_<br>var request = WebRequest.CreateHttp(url);<br>request.Headers[HttpRequestHeader.Range] = $&quot;bytes={start}-{end}&quot;;<br>using (var response = await request.GetResponseAsync())<br>{<br>  var part_length =<br>    long.Parse(response.Headers[HttpResponseHeader.ContentLength]);<br>  if (length != (end - start + 1)) // remember, end is _inclusive_<br>    throw new Exception(&quot;Oops. Unexpected length of content.&quot;);<br>  ...<br>}</pre><p>To create S3 upload parts from specific ranges we need to obey some <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html">rules for multi-part uploads</a>. Primarily, only the last part can be smaller than 5MB. It’s also notable that we can have no more than 10,000 parts in all. This StepFunction based prototype works well within those bounds.</p><p>One more more implementation detail. The payload passed to the function for downloading and creating each part must include the:</p><ul><li>Source URL</li><li>Multi-part Upload ID</li><li>Part Number</li></ul><p>The part number and upload ID are required by S3’s UploadPart API. The part number is also used to determine the range of bytes to copy (remember, the end byte index is inclusive). With all parts created, the final step is to combine them by calling S3’s CompleteMultipartUpload API:</p><ul><li>Multi-part Upload ID</li><li>List of Part Numbers and associated ETags returned by the S3 UploadPart API</li></ul><p>And that’s it.</p><p>Here are what the timings looked like for downloading the same large files mentioned in the start of this article:</p><pre>Download  MB      ms<br>10MB      10      883<br>100MB     100     1299<br>1GB       1024    3713<br>10GB      10240   36267<br>100GB     102400  see below...</pre><p>Except for the smallest file, where the overhead of transitions in the state machine dominate, we’ve delivered a pretty nice speed up. For the largest file (10GB) the speed-up is a near-linear 5x. That’s what I wanted to see in a prototype.</p><p>With only 5 branches each limited to 5GB (the maximum size of a part) the maximum download is 25GB. To test the 100GB file I expanded the number of branches to 20 and found the download time to be 93,128ms (that’s an effective download speed of ~1GB/s or 8Gbps). Since each branch in the 10GB file case downloaded only 2GB vs. 5GB in the 100GB case, this again represents near-linear scaling — the best that can be hoped for with concurrency.</p><p>How far will this go? To support the full potential of S3 would require 10,000 branches — perhaps that would work, but think other things would start going sideways at that scale. Maybe I’ll find out by looking into dynamically-generating the AWS StepFunctions state machine (with retry and error handling, of course)…</p><p>This prototype has taken us from “it can’t do this” to “rocking the download world” with Lambda and a clear and obvious application of the Fanout concept. In a subsequent article I’ll look at a different fanout pattern and scaling out with recursive Lambda executions — mind the guardrails.</p><p>Want to go further with this? Improve robustness by making the part creation restart-able. S3 has an API to list incomplete multi-part uploads and the parts created so far. So when the state machine is restarted the parts that completed on the previous try can be no-op’d. Caution, though. Think about all the ways corruption of the file might happen and what kind of verification is needed to make sure a set of parts are safe to use to complete the upload.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a11b4ef4788e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/circuitpeople/serverless-large-file-downloads-to-s3-a11b4ef4788e">Serverless, large file downloads to S3</a> was originally published in <a href="https://medium.com/circuitpeople">circuitpeople</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>