<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Júnio Silva on Medium]]></title>
        <description><![CDATA[Stories by Júnio Silva on Medium]]></description>
        <link>https://medium.com/@misterjunio?source=rss-8c9598b6e389------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*__w68wt05hoJJQW6Gdezhw.jpeg</url>
            <title>Stories by Júnio Silva on Medium</title>
            <link>https://medium.com/@misterjunio?source=rss-8c9598b6e389------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:30:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@misterjunio/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Accelerating solutions in AWS with ChatGPT]]></title>
            <link>https://awstip.com/accelerating-solutions-in-aws-with-chatgpt-76978bfe2a0a?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/76978bfe2a0a</guid>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[tls]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Wed, 28 Jun 2023 23:33:08 GMT</pubDate>
            <atom:updated>2023-07-01T14:11:15.318Z</atom:updated>
            <content:encoded><![CDATA[<h4>Specifically, save time on the analysis of load balancer logs with Amazon Athena</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y24Ij7Oklu2ARwSZP_fXnw.png" /></figure><p>Large language models (LLMs) have emerged as powerful tools for accelerating the implementation of solutions in various domains.</p><p>However, when it comes to cloud engineering, I haven’t seen many examples of exactly how tools such as ChatGPT can help shave development time. Or maybe I just hadn’t felt the pull of such a clear example as the one I’m about to present.</p><p>The process of setting up querying of logs produced by an <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html">Application Load Balancer</a> using <a href="https://aws.amazon.com/athena/">Amazon Athena</a> is, to be clear, not overly complex. Between looking around for the right information in the documentation of one and the other service, and tailoring the implementation to my specific needs, it’s the kind of task that might typically take me a couple of hours. With ChatGPT, it took me a whole of about 10 minutes. <em>Find out how</em> 👇</p><p>Taking a step back, here is an outline of the task I had in hands: identify developers making requests to our AWS-hosted API (in <a href="https://aws.amazon.com/fargate/">Fargate</a>, behind an ALB) with a <em>deprecated</em> <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> version — i.e. anything below v1.2, as v1.0 and v1.1 got deprecated in 2021. The goal was to assess the impact of enforcing TLS v1.2+.</p><h4>A very high-level prompt</h4><p>I first tried to tell ChatGPT basically what I said above.</p><blockquote>I need to identify developers making requests to my API, which is hosted in AWS, with a TLS version below 1.2.</blockquote><p>It spit out a lot of information about using CloudTrail. It clearly thought I was talking about AWS APIs and went completely down the wrong track.</p><p>But the beauty of these models is the context they retain in-conversation, which makes it easy to nudge them on to a different track.</p><h4>A little more information</h4><blockquote>My API is running on Fargate behind an ALB.</blockquote><p>This time it correctly presented a viable list of steps to take. For brevity, here are the main headers from the list of steps:</p><ol><li>Enable Access Logs on your ALB</li><li>Analyse the ALB Access Logs</li><li>Filter for TLS versions</li><li>Analyse the results</li></ol><h4>A deeper dig</h4><p>With the high-level instructions in a nice list, it was time to get down to implementation details. I could have googled, but why would I leave the comfort of the chat exchange? I repeated for each line 1–4</p><blockquote>Give me more detail about how to accomplish point [N].</blockquote><p>The first response was a summary of <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html">this AWS documentation</a>. No biggie.</p><p>The second a summary of <a href="https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html">this one</a>. Straightforward.</p><p>For the third I had to go back and forth a little because my request_url looked like <strong>https://[host]?…&amp;devid=XXXXXX&amp;…</strong>, and I needed to extract the unique <em>devid</em>s I wanted in the SQL query. Again, something that might have taken quite a bit of time on googling and experimenting, ready in 2 minutes:</p><pre>SELECT devid, ssl_protocol, COUNT(*) AS request_count<br>FROM (<br>    SELECT<br>        regexp_extract(request_url, &#39;devid=([^&amp;]*)&#39;, 1) AS devid,<br>        ssl_protocol<br>    FROM alb_logs<br>    WHERE ssl_protocol NOT LIKE &#39;%TLSv1.2%&#39;<br>        AND request_url LIKE &#39;%devid=%&#39;<br>        AND ssl_protocol &lt;&gt; &#39;-&#39;<br>) subquery<br>GROUP BY devid, ssl_protocol;</pre><p>The fourth was not a technical step, just the conclusion of the analysis process.</p><p>We were therefore easily able to monitor the developers connecting to our API with a TLS protocol version below the requirement. The insight gathered through ChatGPT felt just as similar as an exchange with any AWS support analyst, only much faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*ete-ITCprn26QgnJV72QFw.png" /><figcaption>Before I close, here is a little diagram, because no AWS technical post is complete without one :D</figcaption></figure><h4>What to look out for</h4><p>Time for the mandatory disclaimer that it’s not always rainbows and butterflies.</p><p>The use case I described fit perfectly into what LLMs are best at: gathering information, especially from patterned data, that is otherwise difficult to aggregate. And this was with GPT-3.5 (which is prone to hallucinate once in a while)!</p><p>Because I have been using ChatGPT extensively, I have run into many different issues, for example one that pops up a lot is it suggesting configuration values in CloudFormation that don’t actually exist. It is essential to be mindful of its limitations and potential risks, and ultimately use it as a tool to <strong>assist</strong> human — hopefully expert — decision-making.</p><p>But if you do, the dividends in saved time and effort can be very significant.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76978bfe2a0a" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/accelerating-solutions-in-aws-with-chatgpt-76978bfe2a0a">Accelerating solutions in AWS with ChatGPT</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Temporarily raise provisioned capacity on AWS DynamoDB]]></title>
            <link>https://awstip.com/temporarily-raise-provisioned-capacity-on-aws-dynamodb-c5ec04722d3c?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5ec04722d3c</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[dynamo]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[events]]></category>
            <category><![CDATA[solution-design]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Wed, 29 Mar 2023 11:53:44 GMT</pubDate>
            <atom:updated>2023-04-03T16:07:21.204Z</atom:updated>
            <content:encoded><![CDATA[<h4>Consider event-based solutions to deal with sudden load peaks</h4><p>If you have a DynamoDB table with mostly predictable access patterns, provisioned mode is the way to go. Adequately select your min/max read capacity units (RCU) and write capacity units (WCU), turn on autoscaling, and you should be safe.</p><p>See the image below for a simple example. During relatively low load-variability periods, the provisioned capacity (in red) will be able to accompany the trends of the consumed capacity (in blue) by following preset tracking configuration, backed by CloudWatch Alarms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7Te4waIt0fGX0EHpiWzEA.png" /><figcaption>DynamoDB provisioning autoscaling</figcaption></figure><p>The preset can be quite conservative because it’s preferable to err on the side of caution (i.e. it’s better to over-provision than under-provision) and well, the fact that AWS then makes marginally more money is just a happy<em> </em>coincidence.</p><blockquote>But what if one of your workflows requires a very large and sudden batch write or update into the same table?</blockquote><p>Typically this type of scenario can be “caught” as an event. It is usually either a recurring job that occurs at specific times, or a process that gets triggered by an external source.</p><p>Here’s a quick summary of a setup I had to work with recently:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*C0p-EHp3ykGXuCEyZEtQig.png" /><figcaption>Current architecture</figcaption></figure><ol><li>External system admins upload data (daily) to an S3 bucket via a portal;</li><li>A message is sent to an import SQS queue;</li><li>An ECS Fargate service processes the data and does batch write/updates to a DynamoDB table;</li><li>Once done, the Fargate service puts a message in a different SQS queue for additional handling downstream.</li></ol><p>We started observing excessive latency on step 3. Looking at the write provisioning metrics for DynamoDB, it was easy to deduce what was going on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-JuLd8yCF_t1ci84PvYFEg.png" /><figcaption>Tracking of provisioned capacity not scaling fast enough</figcaption></figure><p>Conservative as it might be, Dynamo’s provisioning autoscaling is not designed to keep pace with such a rapid surge in traffic. The slowness to increase the provisioned capacity was leading to a large number of requests being throttled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ShKux4t9Qwt_PKqTiQb8A.png" /></figure><p>There are ways to solve the problem if you can make changes to the application code. You could, for example, stagger the inserts in a way that would keep the request rate in check. Conversely, the process might be time-sensitive, in which case you want the updates applied as fast as possible. Or — as my case happened to present — you might simply not be in a position to touch the processing job.</p><blockquote>We needed to take advantage of cloud solutions to get around the issue <strong>purely from an infrastructure perspective</strong>.</blockquote><p>The idea we came up with was to increase the provisioned WCU just before the updates started, and decrease them again after the fact.</p><p>Again, we could not instrument API calls from within the app running on the Fargate service. We decided to turn instead to a more native-cloud approach, with an event-based flow using trusty old Lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*NtOEMdbF3fTjD2LZHR6aHw.png" /><figcaption>Proposed architecture</figcaption></figure><p>Now, a few additional things happen in the process:</p><ul><li>When new data is uploaded to S3, a notification is delivered to a Lambda function (relies on <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html">S3 Event Notifications</a>);</li><li>The Lambda function uses the <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTable.html">UpdateTable API</a> to set a higher threshold of provisioned WCU — based off historical data, we can be pretty confident about a reasonable value to set;</li><li>An EventBridge rule detects when a new message is posted to the export queue, and invokes the Lambda function again;</li><li>The Lambda function uses the same API to reset the provisioned DynamoDB capacity to a lower baseline.</li></ul><p>The main advantage of the solution is its simplicity. The Lambda function code is very straightforward, and does not add maintenance effort. The EventBridge rule is trivial to configure. And there is also no added cost.</p><p>A small drawback is that the SQS API call event may take a little while to find its way to CloudTrail and therefore there’s a window where too much capacity is unnecessariyl provisioned. But that is no worse than the autoscaling taking its time to reduce provisioned capacity anyway.</p><p>Another option would have been to leverage a time-based Step Functions workflow to scale up, wait for a certain amount of time when we could be confident the ingestion would be finished, then scale down. However, that solution wouldn’t scale as well. It would pose problems again should the consumed data set size grow and the processing take longer than anticipated.</p><p>We’re still pushing for a chance to slightly alter the processing application code. A modification as simple as posting a notification to an SNS topic instead of a message directly to an SQS queue could open up an even better design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*SfQNJ8UOcsrecNsSHikO-g.png" /><figcaption>Best-case architecture</figcaption></figure><p>With the depicted approach, <em>both</em> the downstream system (still via SQS to avoid any breaking changes) <em>and</em> the Lambda function could subscribe to the end-of-process notifications, hence avoiding the additional delay of waiting for CloudTrail to trigger the scaling down event.</p><p>As usual, in the cloud there are multiple ways to get things working, and it all comes down to a cross between best practices and the constraints of each particular scenario. Even though the specific events that trigger the Lambda function in the solution I described are an S3 bucket notification, and an EventBridge rule, I trust you could easily mimic the overall design to solve the problem of Dynamo DB’s subpar autoscaling.</p><p>Sadly the solution has not yet been implemented in production (currently undergoing testing of the changes, performed using IaC) so I don’t have a nice graph to compare to the initial one. In any case, as always, if you have faced a similar issue and have other ideas on how to best tackle it, please reach out as I’d love to hear them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5ec04722d3c" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/temporarily-raise-provisioned-capacity-on-aws-dynamodb-c5ec04722d3c">Temporarily raise provisioned capacity on AWS DynamoDB</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Christmas miracle wish: an open-source CloudFormation]]></title>
            <link>https://awstip.com/my-christmas-miracle-wish-an-open-source-cloudformation-8c274796e868?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c274796e868</guid>
            <category><![CDATA[validation]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Tue, 27 Dec 2022 11:06:13 GMT</pubDate>
            <atom:updated>2023-01-02T15:01:56.447Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*7zALiEau1ex9bAL3H7wgfw.png" /><figcaption>All I want for Christmas… is an open-source CloudFormation</figcaption></figure><p>I have been known to go on the occasional rant. AWS CloudFormation validation has been known to be absolutely painful. As a BE and cloud engineer, I often have to use CloudFormation. Bring all three together, and you have a recipe for a raving blog post. I’ll try my best to not let it go too sour.</p><p>We’ll look at some context on the source of pain first.</p><h4><strong>How CloudFormation validates templates and changes</strong></h4><p>How does CloudFormation validate a <strong>template</strong>? It doesn’t <em>really</em>. There is a CLI command you can use to “validate” a template, but <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-validate-template.html">the user guide</a> itself states:</p><blockquote>The aws cloudformation validate-template command is designed to check only the syntax of your template. It does not ensure that the property values that you have specified for a resource are valid for that resource. […]</blockquote><p>Stripped down, you’re looking at a JSON/YAML sort of linter (which for some reason you need to be authenticated to use 🤷‍♂ let’s just leave that aside). Indeed, according to the command, the following is a perfectly valid template:</p><pre>AWSTemplateFormatVersion: 2010-09-09<br>Resources:<br>  rDummyBucket:<br>    Type: AWS::S3::Bucket<br>    Properties:<br>      Yada: yada</pre><p>My rDummyBucket resource has a valid type, but what about its properties? Clearly Yada shouldn’t be “valid”.</p><p>The tool does highlight <em>some</em> errors. For example, if I validate the template</p><pre>AWSTemplateFormatVersion: 2010-09-09<br>Resources:<br>  rDummyBucket:<br>    Type: AWS::S3::Bucket<br>    Properties:<br>      Yada: yada<br>Outputs:<br>  oDummyBucketName:<br>    Value: !Ref rDummy</pre><p>it helpfully complains with</p><blockquote>Unresolved resource dependencies [rDummy] in the Outputs block of the template</blockquote><p>because my resource is named rDummyBucket and not rDummy.</p><p>Another positive is that CloudFormation has <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html">its resource specification</a> published on the documentation. Tools such as the <a href="https://marketplace.visualstudio.com/items?itemName=aws-scripting-guy.cform">VS Code Plugin for CloudFormation</a> use it to highlight deviations from the spec in your editor. There are many shortcomings though, particularly when dealing with parameter “types”, results of intrinsic functions, etc.</p><p>You can also combine spec validation with tools such as <a href="https://github.com/bridgecrewio/checkov">Checkov</a>. They complement property validation by evaluating misconfigurations against <a href="https://cve.mitre.org/">CVEs</a> and the like, but they’ll only find misconfigurations <a href="https://www.checkov.io/5.Policy%20Index/cloudformation.html">they scan against</a>.</p><p>Definitely feels like a more holistic approach to template checking is missing.</p><p>Let’s now assume you already have a stack deployed, and focus on how CloudFormation validates a <strong>change</strong> to the stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*cxcfq_bbNT3iIL4mc0Wcdw.jpeg" /><figcaption>You’re taking quite a big chance with every CloudFormation change</figcaption></figure><p>As a good practice, you should always create a <strong>change set</strong> between the template that is currently deployed, and the one you’re intending to deploy next. Doing so makes CloudFormation compare the state of the deployed stack resources with the intended post-deployment state.</p><p>The process has a few advantages over deploying the new template directly:</p><ul><li>You can review the changes that will be applied to your resources (without losing the list if you navigate away) and execute anytime you wish</li><li>You can calmly check whether the changes require resource replacement, which can help flag unintended modifications</li><li>You can include nested stacks in your analysis</li><li>Some errors will be highlighted and cause the change set creation to fail (e.g. failure to resolve SSM parameters)</li></ul><p>The main limitation is what the last bullet point states: again, only <em>some</em> errors are reported when generating change sets.</p><p>If you have an S3 bucket deployed, go ahead and add a Yada property to it and create a change set. The change set will be created successfully and report it needs to update the bucket resource (no replacement). Yet deployment would fail with the message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*nEWn13xprKoJBsoIQNEHSA.png" /><figcaption>Why, really?</figcaption></figure><p>It boggles me that we should need to go so deep, into deployment, for a template with such an “obvious” mistake to be reported as faulty.</p><h4>My latest frustrating example</h4><p>I recently modified a template that defines an Elasticache <em>Memcached</em> cluster, to have the cache node type as a parameter into the stack, pCacheNodeType. The goal was to enable different node types across environments.</p><p>I updated the template accordingly. I then created a change set to apply the update on one of our test cache clusters, using the new parameter to scale the cluster down, in order to save costs. All successful so far.</p><p>Guess what, CloudFormation didn’t like the deployment. Digging through <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticache-cache-cluster.html#cfn-elasticache-cachecluster-cachenodetype">the documentation</a>, I found:</p><blockquote>[…] Changing the CacheNodeType of a Memcached instance is currently not supported. If you need to scale using Memcached, we recommend forcing a replacement update by changing the LogicalResourceId of the resource. […]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*SyVureKxcqZong-k4atUcg.png" /><figcaption>Me in one of many CloudFormation-driven “I’m out” moments</figcaption></figure><p>What’s annoying is not the fact that the particular update I was trying to make is not allowed. Any error must occur due to technical/business reasons, fair enough. Again, what’s annoying is that I had to go all the way down to the stack rollout for the error to surface. If you’re unlucky enough to be dealing with large stacks and large changes, it can easily mean <em>hours</em> of rolling forward and backward.</p><p>[<em>Side ninja fix</em>] As a useful tip for the particular case, instead of following AWS’s advice of changing the LogicalResourceId every time, which is terribly impractical, you can use the trick of adding the node type to the cluster name, forcing CloudFormation to create a new cluster with the updated node type and remove the old one every time:</p><pre># Using the node type in the cluster name because otherwise changing node type fails to deploy<br>ClusterName: !Sub<br>  - cc-${cacheNodeType}<br>  - cacheNodeType: !Join [&#39;-&#39;, !Split [&#39;.&#39;, !Ref pCacheNodeType]]</pre><h4>Thoughts on the future of CloudFormation</h4><p>Although it may well have been a necessary intermediate step towards more advanced infrastructure resource managers, more and more I tend to view CloudFormation as a growingly archaic and clunky tool.</p><p>I never truly understood why the elected term was Infrastructure as <strong>Code</strong>, because I don’t see JSON/YAML as code (regardless of how many add-ons you decorate them with) anymore than I see HTML as code. The move to CDKs feels much more aligned with the blurring of lines between developers and operations.</p><p>BUT, in the meantime there are many, many applications out there based on CloudFormation. The amount of people working with it on the daily is one of the reasons I think it could greatly benefit from becoming open-source. Referring to my frustrating example from before, I would gladly contribute to an open rule engine by adding a validation alert about changing Memcached node types.</p><p>The <a href="https://github.com/aws/aws-cdk">AWS CDK</a> is open-source, as is the recent <a href="https://github.com/hashicorp/terraform-cdk">Terraform CDK</a>. I often wonder what reasons AWS might have to think CloudFormation shouldn’t.</p><p>Perhaps there are product management reasons? Maybe they somehow feel it is too big a chunk of IP to disclose? Note that the CDK still uses CloudFormation under the hood, so you end up hitting the same wall in many scenarios.</p><p>Or they might share my perception that CloudFormation has a (more or less lengthy) expiration date, and can’t justify large amounts of effort put into improving the service. That would be a shame as I have other ideas such as separating the Console visualisation by run — another one the lack of which drives me slightly mad.</p><p>On the technical side, having been a precursor in the space may mean that CloudFormation’s architecture and implementation behind the scenes aren’t the best, and not easy to make public without revealing other important internal mechanisms. Returning to my Memcached cluster example, I note that it’s the Elasticache API throwing the error upon deployment, meaning the rule stated in CloudFormation’s documentation is simply not addressed at all within the service boundaries.</p><p>My hope would be that there exist outer layers, with encoded rules for what transactions are allowed, that can be decoupled. Such a rule engine could be the part made public, benefitting from community contributions to enhancing it. It could act on change set creation and validate all properties of all resources, sketching a downward curve of deployment errors and engineer frustration.</p><p>Okay, I said I wouldn’t let the post go sour and I feel like it’s on the verge so I’ll stop here. I hope I was able to get my point across, and I would love to hear your thoughts. Have you had frustrating experiences with CloudFormation and have ideas on how to improve it? Do you feel like an open-source solution would allow you to contribute to said improvements?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c274796e868" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/my-christmas-miracle-wish-an-open-source-cloudformation-8c274796e868">My Christmas miracle wish: an open-source CloudFormation</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How many different ways can you host a Python web server in AWS?]]></title>
            <link>https://medium.com/@misterjunio/how-many-different-ways-can-you-host-a-python-web-server-in-aws-118ab6fc5862?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/118ab6fc5862</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Fri, 30 Sep 2022 10:07:10 GMT</pubDate>
            <atom:updated>2022-09-30T10:13:13.556Z</atom:updated>
            <content:encoded><![CDATA[<h4>Spoiler alert: I count <em>at least</em> <strong>12</strong></h4><figure><img alt="Traffic signs pointing to different AWS service icons" src="https://cdn-images-1.medium.com/max/1024/1*9nxx1ugV4ibXgbxQJsf4Fg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@bdchu614?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Brendan Church</a> on <a href="https://unsplash.com/">Unsplash</a> (with a little extra AWS flavour)</figcaption></figure><p>The proliferation of cloud services in the past decade has led us to a stage where, whatever the technical challenge may be, there’s usually a panoply of different approaches which can solve it, one way or another.</p><p>Granted, some will definitely be more appropriate depending on the concrete use case, but still, the sheer amount of options can be overwhelming.</p><p>Need an example? You’ve come to the right place. A while back a data scientist mate of mine came to me with a question:</p><blockquote>I have developed a small web application in Python (specifically with the <a href="https://flask.palletsprojects.com/en/2.2.x/">Flask</a> framework). How can I share it on the Internet?</blockquote><p>Simple enough question as it is, it got me thinking of all the different possibilities that AWS alone offers. I will touch briefly on every possible answer I found, listing a few <strong>opinionated</strong> considerations to keep in mind for each of them. <a href="https://www.nosuchthingasafish.com/">And in no particular order, here, we, go</a>.</p><figure><img alt="A puzzled-looking man stares at a laptop surrounded by AWS service icons" src="https://cdn-images-1.medium.com/max/1024/1*ZwhuGyVo1a4HS2msJJWG_A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brucemars?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">bruce mars</a> on <a href="https://unsplash.com/">Unsplash</a> (with a little extra AWS flavour)</figcaption></figure><h3>EC2</h3><p>Starting with EC2. The more traditional and, surprisingly or not, one of the most complex. With EC2 you get full control over fleet configuration and all the work that goes with it.</p><p>In its simplest form that could mean a single small public instance running the Flask app in a process and then something like <a href="https://www.nginx.com/">NGINX</a> to front it. On a more complex setup, you’d have the bells and whistles of a load balancer, autoscaling group, proper public/private network separation and so on.</p><p>With the movement to democratise and simplify workloads by leveraging containerisation in full swing, EC2 would probably not be top of mind to get something off the ground quickly.</p><p><strong>Pros</strong>: Tried and tested. Full control and flexibility to support a myriad of scenarios.</p><p><strong>Cons</strong>: A whole lot of configuration. Not sexy (for lack of a better word) for a developer looking to experiment with cutting edge tech.</p><h3>ECS (EC2)</h3><p>Cue Docker-based offerings. As proof that Docker does <strong>not</strong> magically make everything simpler, running an EC2-backed ECS cluster adds an extra layer of p̶a̶i̶n̶ indirection to what I described for vanilla EC2, with not much additional value.</p><p>Atop a base EC2 setup, you now get to manage task definitions, a container service, and running tasks, with even more complicated networking wires. The tasks cannot have a public IP address, which forces you to have an ALB (or NLB if you’re rich) to make your app reachable. This is an extremely good option to explore if you hate life.</p><p><strong>Pros</strong>: Full control and flexibility to support a myriad of more advanced scenarios. And hey, now you’re using Docker!</p><p><strong>Cons</strong>: A whoooole lot of configuration. Not simple to create something simple.</p><h3>ECS (Fargate)</h3><p>The Fargate flavour of ECS is honestly great (albeit slightly more expensive in general).</p><p>Yes, you still have to get closely acquainted with a few concepts, but it’s smooth sailing once you do. The big plus is that you can easily scale to an actual production-ready system without much hassle, and such a bridge is extremely important.</p><p>For a minimal Flask app as my mate’s intended use case, an ECS task can be assigned a public IP address to target it directly without additional infrastructure. Fantastic.</p><p><strong>Pros</strong>: Smallish learning curve gets you something that’s not too far away from being scalable to a real production application.</p><p><strong>Cons</strong>: A bit pricier. Still requires an initial fair level of understanding of the ECS machinery.</p><h3>EKS</h3><p>Ah.. Kubernetes. K8s if you’re <em>kool</em>. Where to start? I’ll be honest and say that I don’t know the 10th thing about Kubernetes. I’m giving myself a buffer because I do know a few things.. I think?</p><p>I won’t delve into the abyss, will rather just point you to <a href="https://medium.com/bb-tutorials-and-thoughts/how-to-deploy-python-apis-on-aws-eks-using-helm-eb5c15ebb607">this nice Medium article</a> on how to deploy a Flask app into EKS. That should give you a pretty good idea of all the moving parts as well as the breadth of background knowledge required for the endeavour.</p><p>Do not take this route unless you’re very knowledgeable or willing to put in <strong>a lot</strong> of effort. You’ve been advised.</p><p><strong>Pros</strong>: You get to try something that’s super fashionable and future-proof.. I guess that’s a pro?</p><p><strong>Cons</strong>: It’s Kubernetes. Complexity is the biggest down factor.</p><h3>App Runner</h3><p>Some of you may not have heard about App Runner before, but it’s a service that’s seen some nice updates since its launch last year.</p><p>If Fargate is the managed version of ECS, App Runner is basically a sort of “managed Fargate”, meaning you throw it a Docker image and select a couple of spec parameters and off it goes provisioning a properly configured and load balanced container service with a public endpoint.</p><p>If you’re interested in a deeper dive, the <a href="https://cloudonaut.io/">Cloudonaut</a> team have a nice review <a href="https://cloudonaut.io/review-apprunner-simply-containers-on-aws/">here</a> and an equally interesting comparison to Fargate <a href="https://cloudonaut.io/fargate-vs-apprunner/">here</a>.</p><p><strong>Pros</strong>: Simplicity. Not dissimilar to other managed services in the sense that it allows you to focus more on the app and less on its hosting configuration.</p><p><strong>Cons</strong>: Quite a recent service that has naturally yet to achieve full maturity (for example, notably it’s at the time of writing only available in 5 regions).</p><h3>Lightsail</h3><p>Lightsail is AWS’s service for simplifying and speeding up the deployment of simple web apps. Originally intended as a “managed server”, it is now also focused, for the most part, on containers (surprise!).</p><p>Check out <a href="https://aws.amazon.com/getting-started/hands-on/serve-a-flask-app/">this hands-on lab</a> for a quick workshop on how to serve a Flask app on Lightsail containers. You’ll find that it’s a breeze. You may be asking yourself, though:</p><blockquote>How exactly is this different from App Runner?</blockquote><p>Well.. They have different controls, and App Runner is meant as a more scalable service, although for simple public apps they end up addressing the same needs and targeting the same crowds.</p><p><strong>Pros</strong>: Simplicity. Not dissimilar to other managed services in the sense that it allows you to focus more on the app and less on its hosting configuration. 🙃</p><p><strong>Cons</strong>: Only supports basic workloads as it gets hard (and pricy) to scale. Not well integrated with other AWS services.</p><h3>Copilot CLI</h3><p>The Copilot CLI is <a href="https://github.com/aws/copilot-cli">an open-source tool</a> “for developers to build, release and operate production-ready containerised applications on AWS App Runner or Amazon ECS on AWS Fargate”.</p><p>I have to say that in general, I’m not a huge fan of such wrapper tools (not counting <a href="https://medium.com/@misterjunio/standardising-your-mobile-apps-with-aws-amplify-and-react-native-47a0fcbb413a">Tigerbud</a> <em>*coughs*</em>). I think there’s a paradox with trying to simplify things for end users but then proceeding to add so much stuff to make it a complete tool that the level of complexity quickly grows, so the original value of simplicity is lost and we end up with just an extra layer of abstraction/indirection/complexity.</p><p>But hey, talk is cheap and that’s not to denigrate the Copilot CLI at all. Credit where it’s due, it is extremely well executed. It’s as easy to get going as running copilot init and following the prompts. There are then nice utilities to manage environments, pipelines, etc.</p><p><strong>Pros</strong>: Gets you close to a production setup with a handful of commands.</p><p><strong>Cons</strong>: Yet another tool to learn on top of the services it manages for you.</p><h3>Elastic Beanstalk (Python / Docker on EC2 / Docker on ECS)</h3><p>I’m grouping these together because I recognise that it’s somewhat of a cheat to say they are 3 different methods. They all follow Beanstalk’s</p><blockquote>give me the app, I’ll take care of the rest</blockquote><p>premise. Any of them are a legit choice if you’re not that interested in what lies behind your app frontend, as many data scientist types wouldn’t.</p><p>If the app is not readily containerised yet, just go straight with the supported Python platform. If it is, choosing between Docker on EC2 or ECS comes down to whether it’s foreseeable that it might evolve into advanced patterns that justify multi-container orchestration.</p><p><strong>Pros</strong>: Abstracts virtually all the infra complexity and hastily gives you a working app to play with.</p><p><strong>Cons</strong>: Not many, other than that you’re conforming to EB’s mould.</p><h3>Lambda (Python / Docker)</h3><p>Last but not least, our sweet Lambda, which also comes in plain Python and Docker modes.</p><p><a href="https://wsgi.readthedocs.io/en/latest/index.html">WSGI</a>-based frameworks such as Flask are not a great fit for Lambda. That’s likely a generous statement — from my limited Python knowledge, I believe Lambda starts up a separate Python instance on every invocation, and what tools like <a href="https://github.com/zappa/Zappa">Zappa</a>, <a href="https://github.com/slank/awsgi">AWSGI</a> and <a href="https://github.com/logandk/serverless-wsgi">Serverless WSGI</a> do is some magic behind the scenes to circumvent the need for an actual WSGI server.</p><p>I do have a very positive impression of Zappa. It supports a lot of advanced use cases and its core functionality of seamless translating registered Flask routes into API Gateway ones is truly enjoyable.</p><p>Recently (since last year) Zappa <a href="https://ianwhitestone.work/zappa-serverless-docker/">also supports Docker</a>. Experience could be improved as you currently have to tweak around with stuff you wouldn’t expect to, but it’s not too painful overall and represents a great addition to the toolset.</p><p><strong>Pros</strong>: The usual benefits of going serverless. For my mate’s use case the <em>free</em> side of it was especially appealing, as it would be for a lot of people.</p><p><strong>Cons</strong>: As mentioned, not the traditional Flask experience, which may take some getting used to.</p><p>As a personal experiment, I played around with some of the options I described by using Terraform and (coming soon, it’s a WIP) GitHub Actions to deploy and manage a sample Flask app, which I thought I’d share in <a href="https://github.com/misterjunio/aws-python-flask-hosting">this repo</a>.</p><p>Keep in mind that all Terraform modules are my own and the bare minimum to make the sample application run in each service. There are, for example, no calls to the Internet or connection to databases, which would certainly complicate the network and security configurations.</p><figure><img alt="A meme that mocks people that claim to implement the same project across languages but it’s only Hello World" src="https://cdn-images-1.medium.com/max/1024/1*qbL1_VFIFNYhzRV-HOU9mA.png" /><figcaption>😬</figcaption></figure><p>Let’s recap the 12 different options we’ve seen:</p><ol><li>EC2</li><li>ECS (EC2)</li><li>ECS (Fargate)</li><li>EKS</li><li>App Runner</li><li>Lightsail</li><li>Copilot CLI</li><li>Elastic Beanstalk (Python)</li><li>Elastic Beanstalk (Docker on EC2)</li><li>Elastic Beanstalk (Docker on ECS)</li><li>Lambda (Python)</li><li>Lambda (Docker)</li></ol><p>You can argue with some of the above. A couple are only slight variations, but I did leave out others such as splitting Lightsail between <em>with</em> and <em>without</em> containers, the different flavours the Copilot CLI allows, not to mention self-managing Docker directly in EC2 (which may sound mad but I’m sure there’s people out there doing it).</p><p>I hope the article is a faithful illustration of the myriad of possibilities that AWS can open up to attack a given problem. You should always consider your specific requirements, weigh pros and cons of every possible solution, then choose your own journey.</p><p>Good luck.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=118ab6fc5862" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS cross-region CodePipeline for CloudFront + WAF v2]]></title>
            <link>https://awstip.com/aws-cross-region-codepipeline-for-cloudfront-waf-v2-2f9f943cc935?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f9f943cc935</guid>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[cloudfront]]></category>
            <category><![CDATA[waf]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Sun, 03 Jul 2022 11:47:37 GMT</pubDate>
            <atom:updated>2022-09-28T12:09:50.998Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*VBOsXzbIX2mwptFz70ONyg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mbenna?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Mike Benna</a> on <a href="https://unsplash.com/">Unsplash</a> (with a little extra AWS flavour)</figcaption></figure><p>Back in November 2019, AWS released the <strong>second version of their WAF</strong> (Web Application Firewall) service. You can read all about why you should use it (or migrate to it) over version 1 in <a href="https://docs.aws.amazon.com/waf/latest/developerguide/waf-migrating-why-migrate.html">the official AWS documentation</a>.</p><p>One limitation of the new version is that, in WAF v2, Web ACLs (Access Control Lists) that are <strong>targeted at CloudFront distributions can only be created in the </strong><strong>us-east-1 (N. Virginia) region</strong>. If your infrastructure deployment pipelines are set up in a different region, including the WAF resource in those pipelines presents a challenge.</p><p>The solution scaffolded below uses a <strong>cross-region pipeline in CodePipeline</strong> to deploy the WAF v2 to us-east-1, and a <strong>Lambda-backed custom resource in CloudFormation</strong> to fetch the ARN of the Web ACL, in order to attach it to the CloudFront distribution.</p><p>If you like to experiment and get your hands dirty, check out <a href="https://github.com/misterjunio/cloudfront-waf-v2-owasp-top-10">my GitHub repository that hosts a sample implementation</a>. There are detailed instructions on how to get it up and running, so you can dive right in. You’re welcome to stick around or refer back here for a deeper explanation.</p><h3>Here comes the juice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pg9IlAY0BmodRPordf_QjQ.jpeg" /><figcaption>Solution diagram</figcaption></figure><p>The diagram above captures a schematic of the solution. It comprises:</p><ul><li>A CodePipeline cross-region pipeline that deploys <strong>3 CloudFormation stacks</strong>: CI/CD, WAF and another arbitrarily named Main;</li><li><strong>The </strong><strong>CI/CD stack, in </strong><strong>ap-southeast-2</strong>, manages a CodeCommit repo (which triggers the pipeline), and the pipeline itself;</li><li><strong>The </strong><strong>WAF stack, in </strong><strong>us-east-1</strong>, manages a WAF v2 with a Web ACL;</li><li><strong>The </strong><strong>Main stack, in </strong><strong>ap-southeast-2</strong>, manages an S3 bucket as the origin of a CloudFront distribution, and a Lambda custom resource that fetches the ARN of the ACL in us-east-1, so it can be attached to the CloudFront distribution;</li><li>All relevant IAM resources, such as roles and policies.</li></ul><p>Let’s have a closer look at each of the CloudFormation stacks — the solution backbone.</p><h4>CI/CD stack</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHQUJz34q5VSmsGJtE8IQA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@othentikisra?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">israel palacio</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>With the best practices of CI/CD (Continuous Integration / Continuous Deployment) and IaC (Infrastructure as Code) in mind, <a href="https://github.com/misterjunio/cloudfront-waf-v2-owasp-top-10/blob/main/cicd.yaml">the CI/CD stack</a> contains the <strong>definition of the solution pipeline itself</strong>. The pipeline deploys the CI/CD and WAF stacks in parallel before moving on to the Main stack deployment.</p><p>The most interesting aspect is that it is <strong>cross-region</strong>. Note the 2 separate artifact stores, which store the backing files for the CloudFormation resources in each region. I drew the method from <a href="https://medium.com/u/1dcfc4ef0b81">Tejasri Swaroop Boppana</a>’s post <a href="https://medium.com/@tejasriboppana22/cross-region-aws-codepipeline-deployment-to-a-cloudformation-stack-using-aws-cloudformation-95cc451be91c"><em>Creating AWS CodePipeline through AWS CloudFormation for cross region deployment to a CloudFormation stack</em></a> (go check it out and clap it).</p><h4>WAF stack</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*R_BCkehSwyrOV8Ua" /><figcaption>Photo by <a href="https://unsplash.com/@flyd2069?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">FLY:D</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>If you’re not familiar, <a href="https://owasp.org/www-project-top-ten/">OWASP Top 10</a> is a de facto standard, updated every few years, for categorizing the most common web application exploit risks.</p><p>Fellow AWS Ambassador <a href="https://medium.com/u/d4f5176bf028">David Krohn</a> has <a href="https://globaldatanet.com/tech-blog/owasp-top-10-mapped-to-aws-managed-rules">a nice post in the global dataset blog</a> about identifying which AWS Managed Rule Groups can defend against each of the security threats in the latest 2021 edition.</p><p>Even though having <strong>managed rules in WAF is great and recommended</strong>, there are cases in which you may need <strong>more fine-grained control over specific protection rules</strong>. An example I have encountered in the past is performing Attestation for Android devices via <a href="https://developer.android.com/training/safetynet/attestation">Google’s SafetyNet API</a>. It generates an extremely long JWT token you may need to send as an HTTP header to your backend service. In that scenario, you’d want to introduce your own WAF rule to accommodate the larger-than-usual header size.</p><p>In GitHub, you’ll find an <a href="https://github.com/amazon-archives/aws-waf-sample/tree/master/waf-owasp-top-10">AWS-provided CloudFormation template that creates <strong>custom</strong> Web ACL rules</a> that cover the OWASP Top 10. Sadly, it has been archived because it uses <strong>Classic WAF</strong> (v1).</p><p>As part of my overall solution, I decided to grab the template and <strong>update it for WAF v2</strong>. The result was the <a href="https://github.com/misterjunio/cloudfront-waf-v2-owasp-top-10/blob/main/waf-v2-owasp-top-10.yaml"><strong>waf-v2-owasp-top-10.yaml</strong></a> file. To simplify things a little, I restricted it to a <strong>global scope option</strong> (i.e. having only CloudFront as a possible target). If you wish to target local scopes such as an ELB, it should be easy enough to extend/adapt the template.</p><p>Note that the template can be used standalone. For our purposes, it is the definition of our WAF stack.</p><h4>Main stack</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@seanstratton?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Sean Stratton</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>While the first two stacks are sort of supporting material, <a href="https://github.com/misterjunio/cloudfront-waf-v2-owasp-top-10/blob/main/main.yaml">the Main stack</a> represents what would, in a real-world scenario, define your <em>main</em> resources.</p><p>For our example, I decided to stick to a <strong>CloudFront distribution backed by a sample S3 bucket</strong> only. Naturally, there’s no limit to what you could add alongside them in your contexts.</p><p>The assignment of the Web ACL from the WAF stack to the CloudFront distribution is wired through <strong>a Lambda-backed custom resource</strong>. Arbitrarily written in Python, its code is added inline on the CloudFormation template for simplicity. It <strong>tries to fetch the global Web ACL by name and uses the handy </strong><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-lambda-function-code-cfnresponsemodule.html"><strong>cfn-response module</strong></a><strong> to return it to CloudFormation</strong>, or a failure if something goes awry.</p><h3>In closing</h3><p>There are a couple of notes I want to highlight:</p><ul><li>Should go without saying, but please do<strong> not take the GitHub sample implementation as a fully production-ready example</strong>. I was definitely <em>not</em> thorough in ensuring that things like IAM policies follow least-privilege principles, as that was beside my point of focus.</li><li>The CI/CD and Main stacks don’t need to be in ap-southeast-2, <strong>they can be in any region other than </strong><strong>us-east-1</strong>. Sydney just happens to be where I am based.</li></ul><p>I hope the explanations in the article are helpful and ease the use of the multiple parts of the sample implementation.</p><p>As with a lot of things in AWS that keep evolving over time, WAF v2 may eventually come to support being defined in any region even for CloudFront distributions, which would render this whole endeavor a bit pointless. Nevertheless, the patterns and techniques — cross-region pipeline, custom OWASP Top 10 WAF stack, Lambda-backed custom CloudFormation resource — can be recycled for many other purposes.</p><p>Thanks to <a href="https://medium.com/u/61575db35d8">Moha Alsouli</a> for the inspiration for the post and, as always, any comments or feedback are more than welcome!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f9f943cc935" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-cross-region-codepipeline-for-cloudfront-waf-v2-2f9f943cc935">AWS cross-region CodePipeline for CloudFront + WAF v2</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Containerised Lambda: a nice use case]]></title>
            <link>https://awstip.com/containerised-lambda-a-nice-use-case-a8e5133699ac?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8e5133699ac</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Tue, 24 May 2022 03:19:51 GMT</pubDate>
            <atom:updated>2022-09-28T12:07:30.460Z</atom:updated>
            <content:encoded><![CDATA[<h3>Containerized Lambda: a nice use case</h3><h4>An example of how a Lambda container image can help standardize service deployments</h4><h3>The background</h3><p>Ever since <a href="https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/">the inception of Lambda support for container images</a>, I hadn’t encountered a specific need for them, and I had been wondering how much added value the acclaimed feature brought. <em>Why not stick to the fit-for-purpose </em><a href="https://aws.amazon.com/ecs/"><em>ECS</em></a><em>?</em>, I thought.</p><p>Yes, <a href="https://aws.amazon.com/lambda/">Lambda</a> does not support every single runtime out there. Yes, corporate guidelines may mandate total control over the running environment. But these seemed like fringe requirements to me. Obviously, I hadn’t read the announcement with enough attention, especially where it says:</p><blockquote>…if <strong>you’ve invested in container tooling for your development workflows</strong>, it’s not easy to use the same approach to build applications using Lambda.</blockquote><p>Today I want to tell you about the recent experience that finally made this value of containerized Lambdas clear to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*JQZHWSitx7RW-ryjbCGVzQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@frankiefoto?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">frank mckenna</a> on <a href="https://unsplash.com/">Unsplash</a> (with a little extra Lambda flavour)</figcaption></figure><h3>The situation</h3><p>I was working on a project with a backend <strong>based on microservices</strong> (written in C#.NET and hosted in <a href="https://aws.amazon.com/ecs/">ECS</a>). The one <strong>outstanding piece is a Lambda function</strong>, behind an <a href="https://aws.amazon.com/api-gateway/">API Gateway</a>, to <strong>mock third-party dependencies in the </strong><strong>DEV environment</strong> — if you’re curious, that is because the development environments of the external systems are not stable, hence we need a way to ensure that end-to-end tests can run reliably against a set of known static responses. Let’s call it the “mock service” for simplicity.</p><p>All the microservices were deployed via the same (parametrized) <a href="https://www.jenkins.io/">Jenkins</a> job, except for the Lambda function, which followed its own separate deployment methodology.</p><p>We were starting to feel the need for <strong>additional development and testing environments</strong>, to enable parallel feature development, testing of hot-fixes, etc. The core infrastructure team worked on some updates to enable spinning up an entirely new environment for all services. For it to truly be what we called <strong>a “full-stack” deployment</strong>, it had to include the mock service. As such, we had to <strong>rethink its custom deployment method</strong> so that it was no longer tied to a single environment. Knowing that Lambda now supported container images, we figured it was a great time to bring the mock service into the fold and give it the <strong>same standard deployment workflow</strong> we were preparing for all other services.</p><h3>The solution</h3><p>The deployment workflow that we were able to achieve for <strong>all</strong> services was the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*1plM27LyCwdd9mKAOmRa9w.jpeg" /><figcaption>Standardised deployment workflow solution diagram</figcaption></figure><ol><li>Pushes to specific branches in <a href="https://bitbucket.org/product">Bitbucket</a> trigger the Jenkins build job.</li><li>The build job builds the service and packages it up as a <a href="https://www.docker.com/">Docker</a> image that it pushes to ECR. (Note that this job is service-specific.)</li><li>The build job launches a centralized, shared deploy job. The two main jobs are therefore decoupled.</li><li>The deploy job makes use of <a href="https://www.terraform.io/">Terraform</a> templates to handle the infrastructure.</li><li>The Terraform setup is responsible for creating an updated ECS task definition as well as forcing a new ECS service deployment.</li><li>The new ECS task definition points to the latest ECR image of that service, which ensures we’re running the up-to-date code.</li></ol><p>The main point is that <strong>without the Lambda support for containers, using the above strategy would not be possible</strong>. The Lambda function we currently maintain (along with any future ones we may need) would require a specialized treatment, adding complexity to the whole solution and our goal of full-stack environment spin-up.</p><p>Naturally, a bunch of complexity is omitted in the diagram, for instance with respect to how Docker tags and other parameters are used to correctly invoke the deploy steps. The detail on how to build a container image that Lambda can actually run (step 2. above) is also out of scope for this article. If your service is based on .NET, I’ll refer you to <a href="https://docs.aws.amazon.com/lambda/latest/dg/csharp-image.html">the AWS documentation</a>.</p><h3>The wrap</h3><p>Even though I probably still wouldn’t directly opt for Lambda with containers over ECS if standing up a new project, after getting the experience of using a containerized Lambda to standardize the deployment of a microservices-based solution, I am now much more aware of the types of use cases that warrant it.</p><p>After all, with today’s proliferation of products based on containers, it’s great to know that even a fully serverless offering such as Lambda is able to blend in seamlessly with the container landscape.</p><p>Have you encountered other use cases that are simplified by the Lambda + containers combination? If so please reach out, I’d love to hear them.</p><p><em>EDIT: since I published this post, I have implemented an example solution, albeit with different technologies, in </em><a href="https://github.com/misterjunio/lambda-dotnet-minimal-api"><em>this repository</em></a><em> — make sure you give it a check</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8e5133699ac" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/containerised-lambda-a-nice-use-case-a8e5133699ac">Containerised Lambda: a nice use case</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Be careful with the AWS Pricing Calculator]]></title>
            <link>https://medium.com/@misterjunio/be-careful-with-the-aws-pricing-calculator-fe414d4cf0f?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe414d4cf0f</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[rds]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[pricing-calculator]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Fri, 11 Feb 2022 03:39:36 GMT</pubDate>
            <atom:updated>2022-02-11T03:39:36.858Z</atom:updated>
            <content:encoded><![CDATA[<h4>A cautionary tale for differences between regions</h4><figure><img alt="AWS Pricing Calculator" src="https://cdn-images-1.medium.com/max/1024/1*Q-kD3bzcy8AK-hTCCV9TaA.png" /></figure><p>A few weeks ago I was asked to come up with an estimated costing for a given AWS architecture. The intent was twofold: first validate that the minimum client specifications were met by the proposed solution; then optimise the cost as much as possible within the constraints.</p><p>The intended target region was <strong>us-west-1 (N. California)</strong> — that is important. Looking at the plan, it was clear that the main chunk of the cost would be RDS for SQL Server. Using the <a href="https://calculator.aws/#/">AWS Pricing Calculator</a>, I came up with an estimated price for the production database, taking advantage of <a href="https://aws.amazon.com/rds/reserved-instances/">RDS Reserved Instances</a>:</p><figure><img alt="us-west-1 DB cost, paid monthly" src="https://cdn-images-1.medium.com/max/1024/1*LdPIrSxQY-B1jAOCE9duKg.png" /><figcaption>us-west-1 DB cost, paid monthly</figcaption></figure><p>The business was actually OK with paying upfront in order to realise further savings, so happy me adjusted the estimation and reached a nice ~20k USD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y67OCVIPAo4dc13PJPd8FQ.png" /><figcaption>us-west-1 DB cost, paid 1-year upfront</figcaption></figure><p>Fast forward a couple of weeks. By mere chance I happened to explore the possibility of basing the services on the <strong>us-west-2 (Oregon)</strong> region instead. When I went back to the calculator, I was flabbergasted to find that the price estimate for the exact same DB specs was severely higher in us-west-2, from ~2k to ~3.3k if paid monthly, with almost a $14k overall delta from my original upfront cost:</p><figure><img alt="us-west-2 DB cost, paid 1-year upfront" src="https://cdn-images-1.medium.com/max/1024/1*ZdnZQDcD9-oX2TMMyEItgQ.png" /><figcaption>us-west-2 DB cost, paid 1-year upfront</figcaption></figure><p>Yikes. After a bit of investigation and fiddling around with the different configuration parameters in each region, I came to understand that the problem was a conjugation of the <strong>Multi-AZ</strong> setting with the particular instance type I selected. For reasons that still beat me to this day,</p><blockquote>for certain instance types<em>, it is not possible to purchase </em><strong><em>reserved Multi-AZ instances</em></strong><em> in us-west-1.</em></blockquote><p>Indeed, navigating on the AWS Console to the RDS service and initiating the process of purchasing reserved DB instances in N. California doesn’t let you choose any offering type for a Multi AZ deployment option, and hence prevents you from proceeding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fmbx04fDD6DJ5f4FswjfBA.png" /><figcaption>purchasing reserved DB instances in us-west-1</figcaption></figure><p>The same options are available as expected in other regions such as us-west-2. Since running into this fact, I’ve explored multiple different settings to understand what could be the problem. For example, to my huge surprise, the <em>db.m5.8xlarge</em> and <em>db.m5.16xlarge</em> types seem to be accepted, however no other M5 family ones are. <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_SQLServerMultiAZ.html">This AWS documentation</a> presents some limitations of Multi AZ SQL Server, but nothing relevant to my use case.</p><p>While I’m sure that there is probably a plausible explanation that I cannot see — please reach out and let me know if you have it! — , I’ve come to accept the limitation and be glad that I caught it relatively early.</p><p>We all know the silly mistakes that time pressure can lead us to, but in hindsight I should definitely have been more diligent in comparing on-demand instance prices to reserved ones and sensing the too-big-a-discount, or comparing to a different region from the start to flush out any such issues. If there’s a main takeaway here is that the best way to ensure your price estimations are accurate is to take some time to map out different scenarios such as the ones above, and where possible go through the console to ensure the options we want are available.</p><p>That may just avoid you putting yourself in $14k-shaped holes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe414d4cf0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Standardising your mobile apps with AWS Amplify and React Native]]></title>
            <link>https://medium.com/@misterjunio/standardising-your-mobile-apps-with-aws-amplify-and-react-native-47a0fcbb413a?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/47a0fcbb413a</guid>
            <category><![CDATA[amplify]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[concentrix]]></category>
            <category><![CDATA[tigerspike]]></category>
            <category><![CDATA[react-native]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Wed, 29 Sep 2021 02:21:28 GMT</pubDate>
            <atom:updated>2021-10-05T22:44:46.419Z</atom:updated>
            <content:encoded><![CDATA[<h4>Getting started building mobile applications has never been simpler.</h4><p>In the world of tech today there appear to be about a gazillion ways of doing things, and it’s easy to get lost amid all the buzzwords floating around. This situation can be particularly painful if you’re working for an organisation that has several ongoing projects, with multiple teams having multiple ways of working. It’s hard to promote consistency between projects and foster a shared knowledge base among your engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6C4nSvy2Woe0m7bWEn4BA.png" /><figcaption>React Native is here to stay</figcaption></figure><p>This post is a (technical) suggestion of how to alleviate some of these woes, leveraging well-established modern technologies such as <a href="https://aws.amazon.com/amplify/">AWS Amplify</a> and <a href="https://reactnative.dev/">React Native</a>, with a shameless plug at the end into our very first, recently published open-source project at <a href="https://tigerspike.com/"><em>Concentrix Tigerspike</em></a>: <a href="https://github.com/tigerspike/tigerbud-starter"><strong>Tigerbud</strong></a>.</p><blockquote>Side note: React Native is in fashion these days and seems to be here to stay. Without getting into the whole native vs. hybrid app debate that RN kinda tries to eclipse (there are enough online discussions to follow along), more and more people and companies are adopting RN as a real alternative to “traditional” native apps.</blockquote><p>There are plenty of resources to help you navigate the intricacies of the setup with Amplify and React Native (including <a href="https://docs.amplify.aws/start/q/integration/react-native/">Amplify’s own tutorial</a>). React Native tooling can be especially tricky; a small price to pay for the awesomeness you end up with.</p><p>What I really want to shine a light on is yet another example of AWS brilliance. That’s Amplify’s subtle but powerful idea behind the --app flag of the <a href="https://docs.amplify.aws/cli/start/workflows/#clone-sample-amplify-project">amplify init command</a>: with it you can take any project hosted on <a href="https://github.com/">GitHub</a> and, via a single command, make it a template, a blueprint for further development. But I’m getting ahead of myself.</p><p>First, let’s quickly walk through the simplest of examples. Again, there are a ton of resources to help you with installing dependencies and troubleshooting any issues with them, so I’ll just assume you already have a recent version of <a href="https://nodejs.org/en/">Node</a> (and <a href="https://www.npmjs.com/">NPM</a>), as well as the <a href="https://github.com/aws-amplify/amplify-cli">Amplify CLI</a>, ready to go. We’ll be using <a href="https://github.com/expo/create-react-native-app">create-react-native-app</a> to bootstrap a React Native sample application. Navigate on your terminal to a folder of your choosing and run:</p><pre>npx create-react-native-app --use-npm</pre><p>Enter y if you’re prompted to install the create-react-native-app package (also, make sure your project path doesn’t include spaces or slashes in folder names, as that may cause weird errors). Then choose a name for your app and the default template. Your output should look like this (my app name is how-great-is-amplify-rn):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25ec944bd0ade7e573caeb720083ac65/href">https://medium.com/media/25ec944bd0ade7e573caeb720083ac65/href</a></iframe><p>If you want to have a peak at your sample application in the browser, go ahead and run npm run web, entering y if you’re prompted to install the Expo CLI.</p><p>Now, this point is when you would add all the common parts, either new or existing — custom styles, custom components, etc. — for building mobile apps and that you want all of your projects to share.</p><p>Turning into the infrastructure side, let’s initialise the AWS connection with:</p><pre>amplify init</pre><p>Give your project a name (I went with HowGreatIsAmplifyRN), accept the default configuration and authenticate with the AWS profile you should’ve previously set up during the Amplify installation. This will initialise the Amplify project in AWS, setting up a few CloudFormation stacks with some IAM roles and an S3 bucket for deployments.</p><p>From here, you can add cloud functionality as you see fit. For example, most apps require some sort of authentication, which you can easily add with the amplify add auth command. They also normally connect to a backend API, for which there is amplify add api. Follow the <a href="https://docs.amplify.aws/lib/q/platform/js/">respective guides</a> for how to wire up the services in your app code.</p><blockquote>Note that most Amplify infrastructure service resources (e.g. Cognito user pools, AppSync GraphQL APIs, Lambda functions, etc.) are serverless and follow the pay-as-you-go pricing model. As long as you don’t start bombarding your system with many requests, you can remain under the <strong>free</strong> umbrella. One less reason not to give all of it a try.</blockquote><h4>Setting the standard</h4><p>Let’s go back to the standardisation idea. Once you have the foundation you want all your projects to be built from, you can store that foundation in a GitHub repository. After that, you can start taking advantage of Amplify’s templating capability, <a href="https://aws.amazon.com/blogs/mobile/amplify-cli-adds-scaffolding-support-for-amplify-apps-and-authoring-plugins/">introduced a couple of years ago</a>. It’s really as simple as a single command:</p><pre>amplify init --app [your-GitHub-project-url]</pre><p>This command will download your foundation, automatically configure an Amplify environment that mirrors all of its cloud resources, and connect the mobile application to those resources. In sum, all the heavy lifting is pretty much taken care of, and you end up with a ready-to-roll copy of your centralised project which you can begin tailoring to specific requirements.</p><p>Go ahead and run amplify console, select the Amplify console option and marvel at your AWS resources. That’s that!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOCnHmd-XVr1ciV4qLdV1w.png" /><figcaption>How great is Amplify?</figcaption></figure><p>Just beware of a couple of considerations:</p><ol><li>Think about maintainability. Although it’s simple enough to add and remove components, either on the FE or BE, a number of the involved technologies (e.g. the Amplify CLI) are being updated at breakneck speed, so keep an eye out for breaking changes.</li><li>There is a clear limitation: at the moment there is no easy way to introduce new features of your template in a project <em>after</em> it has been initialised. You can hone those features over time but they can only be applied from the start of new projects, not backfilled into previous ones.</li></ol><h4>Ok, I get the idea… but where do I go from here?</h4><p>Depending on your (or your team’s) technical prowess, you may or may not have the capability to build the app components I talked about, or to link them to the infrastructure apparatus.</p><p>If you do, you could argue that all you ended up with is a glorified template. I would counter that to do so in virtually no time and in a way that is repeatable is no mean feat.</p><p>If you don’t, you might be a bit overwhelmed and thinking “heck, it would be nice if there was an off-the-shelf template with some pre-built UI and AWS resources”. Personally, I haven’t found many. And that’s the gap we’re trying to fill with the <a href="https://github.com/tigerspike/tigerbud-starter">Tigerbud</a> starter.</p><p>Instead of bits of this and that, with the magic command I mentioned above — amplify init —-app https://github.com/tigerspike/tigerbud-starter — you get a full set of (customisable) app components already linked to some sample BE services. Easier than ever to get started! Make sure you check out the repo for many more details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*dX79mEqkVSwSLy1Pt63BTQ.png" /><figcaption>Tigerbud logo</figcaption></figure><p>In any case, granted: all the above only represents about the first 10 kms of the marathon, so to speak. At Concentrix Tigerspike we are looking at using this sort of technique to standardise and expedite project preparation, cutting down time, effort and costs. On top of that, we have the expertise to transform this “glorified template” into an actual product, following best practices both at the UX/UI level as well as the AWS cloud and mobile engineering.</p><p>Whether you’re a beginner or seasoned veteran, part of a small company or a large enterprise, reach out to us and we can help you with those 30 odd kms left to go.</p><p>Feel free to leave any suggestions either on the comments below or by opening an issue on Tigerbud’s GitHub. Thanks for reading 👌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47a0fcbb413a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 tips for developing with AWS Greengrass and CloudFormation — Part I]]></title>
            <link>https://medium.com/@misterjunio/6-tips-for-developing-with-aws-greengrass-and-cloudformation-part-i-2b5c091e79bd?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b5c091e79bd</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[aws-iot]]></category>
            <category><![CDATA[greengrass]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Thu, 19 Mar 2020 03:19:05 GMT</pubDate>
            <atom:updated>2020-03-19T03:19:05.476Z</atom:updated>
            <content:encoded><![CDATA[<h3>6 tips for developing with AWS Greengrass and CloudFormation — Part I</h3><h4>These might just help you get a jump over some of the hurdles you may encounter using these services</h4><h3>First things first</h3><h4>Some context</h4><p>With the rise of <em>IaaS</em> (Infrastructure as a Service), web/cloud services are in huge fashion these days. If you’re a young Software Engineer like me, it is likely you have been or will at some point be handed down a solution design (or architect one yourself!) for a project that makes use of many of these different cloud services, be them hosted by <a href="https://aws.amazon.com/">AWS</a>, <a href="https://azure.microsoft.com/">Azure</a> or <a href="https://cloud.google.com/">GCP</a>.</p><p>A lot of these services are, honestly put, awesome to work with, as you feel like you’re fiddling with the forefront of technology (seldom stuff you can simply explain to common folk). Well, something you will invariably find along the line is that for the multiplicity of tools you are presented with, in your use cases they will not always play as well with each other as you’d like for one reason or another, and if you’re anything like me (my hunch is most devs are), you’re more fond of actually writing code than “wasting” time with configuration.</p><h4>Enter… this article’s goal</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PwrDOaTogSlOnQP42pcLdg.jpeg" /><figcaption>AWS Logo</figcaption></figure><p>I was recently tasked with developing the IoT part of one of <a href="https://tigerspike.com/">Tigerspike</a>’s projects. Given that we have an AWS partnership, the solution was designed using AWS services (mainly <a href="https://aws.amazon.com/greengrass/">Greengrass</a>, <a href="https://aws.amazon.com/cloudformation/">CloudFormation</a> and <a href="https://aws.amazon.com/lambda/">Lambda</a> for managing the IoT device, the cloud resources and the actual function code).</p><p>Rather than explaining the whole use case and how the IoT piece fit into our architecture (worthy of another article, no doubt), the <strong>goal of this article</strong> is to share with you a combination of practical solutions for challenges I faced throughout the project while using Greengrass and CloudFormation.</p><p>We’ll be diving deep into technical details of miscellaneous technologies which I assume you already have some degree of knowledge about. Let me outline a summary of the issues I will cover, so instead of reading the whole thing you may <em>cherry-pick</em> (yes, I do love <a href="https://git-scm.com/docs/git-cherry-pick">Git</a><em>)</em> the one(s) you are dealing with. Do think of this article almost as a <strong>hands-on guide for the following</strong>:</p><ol><li>Deploying multiple Greengrass groups from a single CloudFormation template;</li><li>Using Lambda functions from a different (Serverless) stack in a Greengrass group;</li><li>Including the Greengrass group deployment in the development pipeline.</li></ol><p>In order not to make this article too lengthy, I will deliver only the above three tips as a <em>Part I</em>, and in case you’re interested you can move along to <a href="https://medium.com/@misterjunio/6-tips-for-developing-with-aws-greengrass-and-cloudformation-part-ii-c39ce1a930be?sk=6bf3a8ddb5ffc68e0e5bd7feca2324ed"><em>Part II</em></a>, which goes through:</p><ol><li>Handling IoT rules and rule permissions from CloudFormation;</li><li>Streamlining the test of code changes in a physical Greengrass IoT edge device;</li><li>Accessing local Greengrass IoT edge device files for simple operations.</li></ol><p>For each of these I will explain how and why the difficulty came about in my experience, and what learnings I needed to resolve the matter. Let’s jump in.</p><h3>The tips</h3><p><em>Note: for brevity I’ll refer to Greengrass as GG and to CloudFormation as CFN throughout.</em></p><h4>Deploying multiple Greengrass groups from a single CloudFormation template</h4><p>Let’s assume you have created a CFN template to handle your GG group (you can follow something like <a href="https://aws.amazon.com/blogs/iot/automating-aws-iot-greengrass-setup-with-aws-cloudformation/">this</a> in order to do so). Let’s also imagine that, as a future-aware developer, you know you will need to deploy groups with that exact same configuration to multiple devices.</p><p>For all GG resource types in CFN, for example AWS::Greengrass::CoreDefinition or AWS::Greengrass::FunctionDefinition, the logical ID (i.e. custom name you give to the resource) needs to be unique within the template, but not between different groups. The actual physical ID (Name property for some resources and Id property for others) is what actually needs to differ from group to group and resource to resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T6tgDY4wfHfCmr2_8agYlw.png" /><figcaption>Logical vs. physical ID</figcaption></figure><p>What does this mean in practice? It means that you can use the exact same template for the various groups, as long as you keep the Name (or Id where applicable) properties of your definition resources dynamic (and unique), which you can do with the help of CFN <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html">Parameters</a>. Suppose that your group only has one IoT <em>thing</em> which is one core. The part of your CFN template that defines it will look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7728f8c51523f75af4409b486f58e4c/href">https://medium.com/media/a7728f8c51523f75af4409b486f58e4c/href</a></iframe><p>Notice how, although our resources’ logical IDs are simply namedCoreThing, CoreDefinition and CoreDefinitionVersion (and will be for every group), the ThingName property of the CoreThing, the Name property of the CoreDefinition as well as the Id property of the Core in the CoreDefinitionVersion get their values by referring to two variables: pEnvName for the environment the group is to be deployed to (usually something likedev, uat or prod); and pCoreId for the core ID (identifier for each core). These would be passed to the template as:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dd2d2a879e0a31d42e99e09d9bcf5c5/href">https://medium.com/media/3dd2d2a879e0a31d42e99e09d9bcf5c5/href</a></iframe><p>You would do something similar to define your functions (see code below). Again, notice the Name property of the FunctionDefinition resource as well as the Id property of the Function resource inside the FunctionDefinitionVersion. This particular GGIPDetector Lambda function is provided by AWS and actually allows the GG group to have its core connectivity set as automatically detectable by default.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de5178e3c5344dae4ef08c22a9b96a7a/href">https://medium.com/media/de5178e3c5344dae4ef08c22a9b96a7a/href</a></iframe><p>What you’d do from here is just guarantee that the pair pEnvName+pCoreId you specify when deploying the CFN template is never repeated for different groups and you’d be good to go with multiple GG groups.</p><blockquote><em>First takeaway: to deploy multiple GG groups from a single CFN template</em>, use that one template with the same logical IDs but using parameters to create unique physical IDs<em>.</em></blockquote><h4>Using Lambda functions from a different (Serverless) stack in a Greengrass group</h4><p>Our CFN template was a little more complicated than mentioned in tip 1. To keep things consistent with the backend system we were developing in parallel, our Lambda functions destined for the IoT device were managed through a <a href="https://serverless.com/">Serverless</a> framework stack.</p><p>The problem is that you need to provide the functions in the CFN AWS::Greengrass::FunctionDefinitionVersion resource through the FunctionArn property:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*br5IG50Pk2V8Q8EXr1D2Dw.png" /></figure><p>Problem: how to get the qualified ARNs of our custom Serverless Lambda functions into this stack? We could hardcode them, but what happens when we deploy a new function version and want this new version to be deployed to the GG core instead of the old one? Should we update the template manually again and again with the new ARN? That’s not very handy.</p><p>This is a common problem with CFN — how to export Lambda qualified ARNs to a different stack. The usual pattern using the <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-importvalue.html">ImportValue</a> intrinsic function does not work well because that is intended for static values. Try it with a qualified Lambda ARN and it will fail on update.</p><p>Once again, our solution was to resort to CFN Parameters. Our development pipeline ran a set of scripts that would, in summary:</p><ul><li>First, deploy the latest Serverless stack (i.e. with the most up-to-date Lambda functions) through the sls deploy command;</li><li>Second, make use of the Serverless CLI’s sls info -v command and some manipulation with the <a href="https://stedolan.github.io/jq/"><em>jq</em></a> tool to output the Lambda ARNs into a JSON file;</li><li>Lastly, deploy the CFN GG groups by feeding in as Parameters the Lambda ARNs read from that JSON file.</li></ul><p>I totally understand if this sounds a bit too abstract, however it’s really as far as I can go with a conceptual idea explanation. Hit me up personally for more implementation details if you need them, or please share if you have found a better way to share dynamic values across stacks!</p><blockquote><em>Second takeaway: to use Lambda functions from a different stack in a GG group, </em>make your pipeline output their qualified ARNs to a file and then feed them into the group template as parameters<em>.</em></blockquote><h4>Including the Greengrass group deployment in the development pipeline</h4><p>For this tip it is important to first clarify the difference between deploying a CFN stack that defines the GG resources and actually deploying the GG group to physical devices. The first simply creates/updates (I like the database-originated term <em>upsert</em>) the GG group, its functions, subscriptions, etc., while the second one “installs” that updated group onto the GG core and/or any other IoT edge devices. To differentiate them I will refer to them correspondingly as “cloud deployment” and “physical deployment”.</p><p>If you’ve been working with GG for a little while, you might have noticed the little “annoying” fact (which to be fair does make sense) that you cannot perform a group cloud deployment if that group is currently in a successful physical deployment state.</p><p>Our development pipeline was not prepared for this and I’ll be honest: it triggered me. I didn’t want to be manually resetting the physical deployments in the AWS Console every time I wanted to deploy a new GG group version to the cloud. The solution: automate the reset of the physical deployments on our pipeline script.</p><p>The AWS CLI provides a method to <a href="https://docs.aws.amazon.com/cli/latest/reference/greengrass/reset-deployments.html">reset group deployments</a> (the physical kind). To use it, I noticed that pretty much all I needed was to somehow get each cloud-deployed group’s ID. Thus the first step was to work with CFN <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html">Outputs</a> to (redundancy aside) <em>output</em> that ID from the CFN template for the GG group, like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8e04803a3185d7d4207b9b94c287d95/href">https://medium.com/media/d8e04803a3185d7d4207b9b94c287d95/href</a></iframe><p>Then in my pipeline script, I was able to use the command <a href="https://docs.aws.amazon.com/cli/latest/reference/cloudformation/describe-stacks.html">describe-stacks</a> with a query to extract that group ID from the CFN stack output and then use the <a href="https://docs.aws.amazon.com/cli/latest/reference/greengrass/reset-deployments.html">reset-deployments</a> command to force physical deployment reset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4762c9749e1e99bc70c9d58fd9a2b25/href">https://medium.com/media/f4762c9749e1e99bc70c9d58fd9a2b25/href</a></iframe><p>This step is performed each time before CFN deployment, ensuring that, if the GG group is already deployed to the cloud, its physical deployments will not get in the way of an update.</p><p>Something worth mentioning was that we decided to keep the physical deployment post cloud deployment as a manual step. This was intentional because</p><ul><li>We might want to update the GG cloud group multiple times before physical deployment;</li><li>The edge device might not be ready;</li><li>There might be extra safety reasons to consider.</li></ul><p>However we could have automated it as well if we wanted to with the <a href="https://docs.aws.amazon.com/cli/latest/reference/greengrass/create-deployment.html">create-deployment</a> command. Furthermore, the group ID output proves useful when you have many devices in production and want to perform bulk physical deployment. If that’s something you need, I’ll point you to <a href="https://docs.aws.amazon.com/greengrass/latest/developerguide/bulk-deploy-cli.html">this AWS documentation</a>, which you can combine with this tip.</p><blockquote><em>Third takeaway: to include the GG group deployment in the development pipeline, </em>output the GG group ID from the CFN template and grab it in the pipeline to reset physical deployments before cloud deployment<em>.</em></blockquote><h4>Wrap around</h4><p>The first part of my development tips ends now. <strong>Tune in to the sequel</strong> at <a href="https://medium.com/@misterjunio/6-tips-for-developing-with-aws-greengrass-and-cloudformation-part-ii-c39ce1a930be?sk=6bf3a8ddb5ffc68e0e5bd7feca2324ed"><em>Part II</em></a> to uncover even more dev tips, if you liked this one and/or if you didn’t find all the answers you were looking for.</p><p>And of course, feel free to drop me a comment with any doubts, questions, suggestions or feedback you might have. I’d love to hear it if you have any improvements over anything I went over in this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b5c091e79bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 tips for developing with AWS Greengrass and CloudFormation — Part II]]></title>
            <link>https://medium.com/@misterjunio/6-tips-for-developing-with-aws-greengrass-and-cloudformation-part-ii-c39ce1a930be?source=rss-8c9598b6e389------2</link>
            <guid isPermaLink="false">https://medium.com/p/c39ce1a930be</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-iot]]></category>
            <category><![CDATA[cloudformation]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[greengrass]]></category>
            <dc:creator><![CDATA[Júnio Silva]]></dc:creator>
            <pubDate>Thu, 19 Mar 2020 03:16:26 GMT</pubDate>
            <atom:updated>2020-05-11T11:45:40.379Z</atom:updated>
            <content:encoded><![CDATA[<h3>6 tips for developing with AWS Greengrass and CloudFormation — Part II</h3><h4>These might just help you get a jump over some of the hurdles you may encounter using these services</h4><h3>Second things second</h3><h4>Continuation from <a href="https://medium.com/@misterjunio/6-tips-for-developing-with-aws-greengrass-and-cloudformation-part-i-2b5c091e79bd?sk=1b4cb55f30491679c1291d42d41b1db8"><em>Part I</em></a></h4><p>If you’re reading this then you know that <em>I am gone by now</em>. Well, you would if this was a cheesy movie. It’s not, so you’re likely to have landed first on <a href="https://medium.com/@misterjunio/6-tips-for-developing-with-aws-greengrass-and-cloudformation-part-i-2b5c091e79bd?sk=1b4cb55f30491679c1291d42d41b1db8"><em>Part I</em> </a>of these development tips. If not and you’d like to know where the motivation for these articles came from, make sure to check that one out.</p><p>Otherwise all you need to know is that in this text I delve into three more development tips for developing with the AWS offerings <a href="https://aws.amazon.com/greengrass/">Greengrass</a> and <a href="https://aws.amazon.com/cloudformation/">CloudFormation</a>. If you’re to understand what you’ll read, I expect you to have some working knowledge of both of those services. Read through or take this article as a reference if you ever run into any of these issues:</p><ol><li>Handling IoT rules and rule permissions from CloudFormation;</li><li>Streamlining the test of code changes in a physical Greengrass IoT edge device;</li><li>Accessing local Greengrass IoT edge device files for simple operations.</li></ol><p>Let’s dive right into it.</p><h3>The tips</h3><p><em>Note: for brevity I’ll refer to Greengrass as GG and to CloudFormation as CFN throughout.</em></p><h4>Handling IoT rules and rule permissions from CloudFormation</h4><p>Let’s say you have a GG group physically deployed to your IoT edge devices. Now you might want to send some information from them back to the cloud (maybe you even already set up a <a href="https://docs.aws.amazon.com/greengrass/latest/developerguide/config_subs.html">GG Subscription</a> for it). Plus, if you’re working with Lambda microservices, it is likely you want to trigger a cloud function in reaction to something published from your devices. Perhaps you have even got to the point where you figured out the way to go is with <a href="https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html">AWS IoT Rules</a>.</p><p>How do you incorporate that with your CFN-based development? By defining those IoT rules in a CFN template, naturally. Let’s have a look at the code for one such example rule:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a65b9c68e720539eebc7221b373d07e4/href">https://medium.com/media/a65b9c68e720539eebc7221b373d07e4/href</a></iframe><p>This rule named devProcessSignalRule triggers on any message that is published into the <em>/dev/+/process/signal </em>MQTT topic (<em>+</em> stands for a wildcard, in this case I wanted all my IoT cores to send a message to their respective topic e.g. <em>/dev/core1/process/signal</em>, but the same Lambda function to handle all the messages in the cloud). It then invokes a Lambda function with an ARN that is imported from another stack (<em>dev-process-signal-function-arn</em> is the export name), passing the message payload as the event function argument. After CFN deployment I was able to go to the AWS IoT service in the AWS Console and indeed see the rule there:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIs6KoDlnGHLZcUgXzcIfQ.png" /></figure><p>Nonetheless, I faced a problem: the rule didn’t seem to call the Lambda function into action. As many messages as I sent to the correct topic and no matter what the message, I did not see any logging on <a href="https://aws.amazon.com/cloudwatch/">CloudWatch</a> nor the effects that I was expecting on my <a href="https://aws.amazon.com/rds/">RDS</a> database. Intriguing.</p><p>The rub had to do, as many times happens while connecting services, with permissions. Specifically, Lambda function permissions. Although my GG group role had permission to publish to the AWS IoT service, the AWS IoT service itself was not whitelisted in my Lambda function as an authorised invoker.</p><p>As per the fix, you can find it below. In the end it was as simple as adding an AWS::Lambda::Permission resource on the same CFN template as the ProcessSignalRule IoT rule above, allowing the rule to trigger the intended Lambda. After deploying the template everything started working nicely. Hopefully this spares you a headache.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d705d35f1f61a588bb04743a38cfef3c/href">https://medium.com/media/d705d35f1f61a588bb04743a38cfef3c/href</a></iframe><blockquote><em>First takeaway: to handle IoT rules and rule permissions from CFN</em>, make sure you add a Permission resource on your template to allow your IoT rule to trigger your target functions<em>.</em></blockquote><h4>Streamlining the test of code changes in a physical Greengrass IoT edge device</h4><p>After my CFN template and cloud deployment for the GG group were all set up (see the previous tips for quite a lot of information about that), my usual development workflow became as follows:</p><ol><li>Modify Lambda functions’ code however suited my needs;</li><li>Deploy the CFN template to the cloud with the updated function definitions;</li><li>Deploy the group to my physical device;</li><li>Test the changes I just made.</li></ol><p>However, having to go through this painstaking process every time I wanted to make even the slightest change to any function soon became tiresome, to the point I felt like there had to be a better way. After dedicating some time to investigate the GG folders living on my physical device, I found out something interesting.</p><p>Inside the IoT edge device, the Lambda functions’ code sits in the folder <em>/greengrass/ggc/deployment/lambda/</em>. For example, the folder <em>/greengrass/ggc/deployment/lambda/</em><strong><em>[example function Lambda ARN]</em></strong><em>/</em> contains the entire code for some example Lambda function.</p><p>Let’s assume you are connected to the device by <em>SSH</em> and imagine you want to test a small change to the file <em>index.js</em> of this Lambda function. But you don’t want to go through the effort of deploying a new GG group version with the updated Lambda to the cloud and then the device. One option is to simply navigate to the mentioned deployment folder on the device and run:</p><pre>vim index.js</pre><p>If you’re familiar with <a href="https://www.vim.org/"><em>vim</em></a>, you can use it to edit the file as you please. If you’re not, you can use another editing tool or instead edit it in your local environment with your favourite editor, but then you need to somehow place it inside the device. In the same scenario, you can use <a href="http://www.scp-wiki.net/"><em>SCP</em></a> on Mac or Linux (or do something similar with <a href="https://www.ssh.com/ssh/putty/putty-manuals/0.68/Chapter5.html">Putty and PSCP on Windows</a>) to pass files to the device by going (in your machine) to the folder where they are located (in this case the folder where <em>index.js</em> is placed) and running:</p><pre>scp index.js <strong><em>[user]</em></strong>@<strong><em>[IP address]</em></strong>:/greengrass/ggc/deployment/lambda/<strong><em>[example function Lambda ARN]</em></strong>/</pre><p>(Obviously <em>user</em> and <em>IP address</em> refer to the same user and IP address you use to SSH into your device.)</p><p>This will override the <em>index.js</em> in the deployment folder of the example function on the device in favour of the new one.</p><p>In both cases (in-place editing and file transfer), GG will not pick up the changes in the code until you restart its daemon (meaning the device will not be executing the new code you want to test). Luckily GG provides an easy way to do that. Simply run these two commands on the device:</p><pre>/greengrass/ggc/core/greengrassd stop<br>/greengrass/ggc/core/greengrassd start</pre><p>Momentarily all the Lambdas should be executing with their new code. If you’re crafty with bash you can even write a script on the device side to automate the whole thing: get all the files you want from your local machine into the device’s correct location, and then restart the daemon, every time you make some changes. Up to you.</p><p>When you’re happy with your outcomes you can get your changes properly deployed through the AWS services, updating the Lambda ARNs in your CFN stack for GG, to make double sure everything still works properly.</p><blockquote><em>Second takeaway: to streamline the test of code changes in a physical GG IoT edge device</em>, make the code changes directly to the deployed files and restart the GG daemon on the device<em>.</em></blockquote><h4>Accessing local Greengrass IoT edge device files for simple operations</h4><p>The beauty of GG is that it runs on containers. Coming from an Electrical / Embedded Engineering background, the fact that the same kind of Lambda function that runs in the cloud can also run in an (albeit very powerful) Linux device just fascinates me.</p><p>For this tip we face the flip side to this beauty, which sometimes is practicality. One of my requirements was that I had to write two Lambda functions that needed access to the local device file system, one to reboot the device and another to upload the GG system file <em>/greengrass/ggc/var/log/crash.log</em> (which for some reason is not synced to CloudWatch logs — follow <a href="https://docs.aws.amazon.com/greengrass/latest/developerguide/greengrass-logs-overview.html">this link</a> for more info) daily to an <a href="https://aws.amazon.com/s3/">S3</a> bucket.</p><p>They seemed like simple enough tasks until I got to actually develop and test them. The difficulty comes from the fact that the GG container bus cannot access the system controller binaries, so I was never able to run the reboot command or get access to the GG deployment files from within a function.</p><p>You can resolve this by carefully inspecting the <a href="https://docs.aws.amazon.com/greengrass/latest/developerguide/lambda-group-config.html">configuration options</a> available to GG Lambda functions. The <strong>Containerization</strong> and <strong>Run As</strong> are of special importance in this case. The first allows you to run the functions as a regular Linux process, while the second lets you choose the UID and GID access identity.</p><p>So if you have some resources in your local system that are not accessible from the container (such as the <em>crash.log</em> file) or that require access from an identity other than the default GG user and group (such as the <em>reboot</em> executable), you can specify them with these options on your GG CFN template function definitions.</p><p>Just be extra careful about following the AWS recommendation not to run functions as the device root user, as that poses a security threat. For something as simple as a reboot though, you can probably get away with it with little to no risk.</p><blockquote><em>Third takeaway: to access local GG IoT edge device files for simple operations, </em>change the <strong>Containerization</strong> and <strong>Run As</strong> configurations accordingly to suit your needs<em>.</em></blockquote><h3>Wrap up</h3><p>I honestly hope that this collection of assorted development learnings I took away from my recent work with AWS Greengrass and CloudFormation are useful for someone out there under a similar situation.</p><p>And of course, feel free to drop me a comment with any doubts, questions, suggestions or feedback you might have. I’d love to hear it if you have any improvements regarding anything I went over in this couple of articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c39ce1a930be" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>