<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Steven Staley on Medium]]></title>
        <description><![CDATA[Stories by Steven Staley on Medium]]></description>
        <link>https://medium.com/@stevenpstaley?source=rss-50d706bb696d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5DAXZDDJFX4hvfizsK83VA.jpeg</url>
            <title>Stories by Steven Staley on Medium</title>
            <link>https://medium.com/@stevenpstaley?source=rss-50d706bb696d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 06:05:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stevenpstaley/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Small Change to Clean Up Your Github Projects]]></title>
            <link>https://blog.museflow.ai/a-small-change-to-clean-up-your-github-projects-4a466825f331?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a466825f331</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[rebase]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 20:25:54 GMT</pubDate>
            <atom:updated>2024-12-27T20:44:19.699Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dgwo5x_Go_3R3Z1n1cBGWw.png" /><figcaption>OpenAI. (2024). Generated using DALL·E.</figcaption></figure><p>Tl;Dr: use “squash and merge”</p><p>Recently, my org migrated to <a href="https://github.com/enterprise/trial">Github Enterprise</a> (the self-hosted version of GitHub), and there was a configuration used for PRs that, in my ignorance, I had never noticed or used in my projects. You may have seen it before (it’s quite visible). Next to the “Merge pull request” button, you’re given a drop-down with a few options: merge, squash/merge, and rebase/merge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZ5BWUAx1NsR9nMIaF05jg.png" /></figure><p>I never clicked that drop-down, always opting to hit “merge pull request”. However, one of these other options is most likely what you’ll want to use. After adopting GitHub Enterprise at my org, I noticed that “squash and merge” was the only option provided.</p><p>After doing a bit of research, I found that you can “effectively make this the default option” from this <a href="https://stackoverflow.com/a/49203410/7885588">stack overflow response</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i19pNoiewX5D6-_Z8izYLQ.png" /></figure><p>By going into general settings, navigating to pull requests, and unchecking the other merge options like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejitAmkPHZWif0ZSHdX73w.png" /></figure><p>From there, I imagine you, like myself, would want your PR description to be the default commit message after the squash. To do that, click the “default message” drop-down and select “pull request title and description” from the options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*QsTmlZp6K_GTpkQofPnmEQ.png" /></figure><p>Now, “squash and merge” will be the default option in your PRs, and the PR title and description will be used as the commit message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxEYjtacpq6M5uloUIjDZQ.png" /></figure><p>This might be old news to you, but to me, it wasn’t, so I thought I’d share. I tend to use git commits more like drafts than a way to save fully functional work, primarily out of fear of losing anything. Many times, I know I should do a rebase before I merge, but doing so with the CLI is always a hassle, and sometimes I don’t bother. This is a clean way to ensure your commit history is clean without additional work.</p><p>Happy holidays!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a466825f331" width="1" height="1" alt=""><hr><p><a href="https://blog.museflow.ai/a-small-change-to-clean-up-your-github-projects-4a466825f331">A Small Change to Clean Up Your Github Projects</a> was originally published in <a href="https://blog.museflow.ai/">The MuseFlow Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Few Unshakable Principles for Writing Better Code]]></title>
            <link>https://itnext.io/a-few-unshakable-principles-for-writing-better-code-2be0aecf4b37?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2be0aecf4b37</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 20:10:26 GMT</pubDate>
            <atom:updated>2024-10-19T17:57:48.098Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5H8pa3sKUrA5q-vEOI_ZQw.png" /><figcaption>OpenAI. (2024). <em>ChatGPT</em> [Large language model]. <a href="https://chatgpt.com/">https://chatgpt.com</a></figcaption></figure><h3>Falling out of love with “Clean Code”</h3><p>Over the past few years I’ve become far less enamered with “clean code”. For a little while, I started to think that it didn’t matter at all. What mattered was solving problems — the faster the better. However, I’ve come to realize that may have been a step too far, and likely the outcome of getting too accustomed to working with skilled engineers who generally don’t require all that much editing.</p><p>After working with a newer engineer for the first time in a while, I realized that there were some driving principles that had subconsciously become part of my coding practice. So, instead of blathering about how clean code doesn’t matter, while hypocritically adhering so some rather strict rules myself, I figured that I would instead talk about them.</p><p>Here are the few, but important, principals of writing clean code that I like to stick to. When reviewing, this is what I look for.</p><p>By the way, I know that talking about clean code can be a can of worms and everything is subjective, so please don’t @ me if you can’t be civil.</p><h3>“My Code Is Self Documenting”</h3><p>When I started my career at a small government subcontractor, it was a joke among the team that one of the senior engineers, in his interview, when prompted to discuss his philosophy on documentation, had said “my code is self-documenting”.</p><p>I think that speaks to how he performed in the rest of the interview and the sense of humor of the interviewer that he got the job. One of the benefits of working for a smaller company.</p><p>However, there was some truth to the idea, and I’m stealing this directly from the book “<a href="https://www.oreilly.com/library/view/clean-code-a/9780136083238/">Clean Code</a>” (which I haven’t actually read and which some have <a href="https://qntm.org/clean">criticized</a>) when I say that “code should read like English”.</p><p>This is as close as you can get to code being “self-documenting” and is a very valid approach to writing readable code. Code “reading like English” is a combination of a few traits covered in Clean Code, namely: functions should do one thing, be well named, and be short (kind of a result of doing one thing). Let’s talk about these concepts in more detail.</p><h3>Functions Should Do One Thing</h3><p>This is perhaps one of the larger tells as to an engineer’s experience that I’ll glean from a PR review, and it is also one of the simplest ways to clean up your code.</p><p>Most of the other concepts in this post will stem directly from this one principle. If your function does one thing, it will be easy to name. If your function does one thing it will be short. If your function does one thing it will have only one return type. If your function does one thing, it will be easier to test. If your functions do one thing, are short, and well named, your code will read like english.</p><p>That’s pretty much it. Let’s look at an example. Here’s a function that does too many things (provided by ChatGPT). Try not to read it for more than 3 or 4 seconds.</p><pre><br>function processOrder(order: any) {<br>    if (!order || !order.items || order.items.length === 0) {<br>        console.log(&quot;Invalid order&quot;);<br>        return;<br>    }<br><br>    let total = 0;<br>    for (let item of order.items) {<br>        total += item.price * item.quantity;<br>    }<br><br>    if (order.couponCode === &quot;DISCOUNT10&quot;) {<br>        total *= 0.9;<br>    }<br><br>    if (order.paymentMethod === &quot;credit_card&quot;) {<br>        console.log(`Processing credit card payment for ${total}`);<br>    } else if (order.paymentMethod === &quot;paypal&quot;) {<br>        console.log(`Processing PayPal payment for ${total}`);<br>    } else {<br>        console.log(&quot;Unsupported payment method&quot;);<br>        return;<br>    }<br><br>    console.log(`Sending confirmation email to ${order.email}`);<br>}<br>function main() {<br>    const order = {<br>        items: [<br>            { name: &quot;T-shirt&quot;, price: 20, quantity: 2 },<br>            { name: &quot;Shoes&quot;, price: 50, quantity: 1 }<br>        ],<br>        couponCode: &quot;DISCOUNT10&quot;,<br>        paymentMethod: &quot;credit_card&quot;,<br>        email: &quot;customer@example.com&quot;<br>    };<br><br>    processOrder(order);<br>}</pre><p><em>OpenAI. (2024). ChatGPT [Large language model]. https://chatgpt.com</em></p><p>In a quick skim, it’s not obvious what it does. Let’s clean this up into multiple short, specific functions:</p><pre><br>function processOrder(order: any) {<br>    // ignore lack of error handling<br>    validateOrder(order)<br>    const total = calculateTotal(order);<br>    const discountedTotal = applyDiscount(total, order.couponCode);<br>    processPayment(order.paymentMethod, discountedTotal))<br>    sendConfirmationEmail(order.email);<br>}<br><br>function validateOrder(order: any): boolean {<br>    if(!order &amp;&amp; order.items &amp;&amp; order.items.length &gt; 0) {<br>      throw new Error(&quot;Invalid order&quot;);<br>    }<br>} <br><br>function calculateTotal(order: any): number {<br>    return order.items.reduce((sum: number, item: any) =&gt; sum + item.price * item.quantity, 0);<br>}<br><br>function applyDiscount(total: number, couponCode: string): number {<br>    return couponCode === &quot;DISCOUNT10&quot; ? total * 0.9 : total;<br>}<br><br>function processPayment(paymentMethod: string, amount: number): boolean {<br>    switch (paymentMethod) {<br>        case &quot;credit_card&quot;:<br>            console.log(`Processing credit card payment for ${amount}`);<br>            return true;<br>        case &quot;paypal&quot;:<br>            console.log(`Processing PayPal payment for ${amount}`);<br>            return true;<br>        default:<br>            throw new Error(&quot;Unsupported payment method&quot;);<br>    }<br>}<br><br>function sendConfirmationEmail(email: string) {<br>    console.log(`Sending confirmation email to ${email}`);<br>}<br>function main() {<br>    const order = {<br>        items: [<br>            { name: &quot;T-shirt&quot;, price: 20, quantity: 2 },<br>            { name: &quot;Shoes&quot;, price: 50, quantity: 1 }<br>        ],<br>        couponCode: &quot;DISCOUNT10&quot;,<br>        paymentMethod: &quot;credit_card&quot;,<br>        email: &quot;customer@example.com&quot;<br>    };<br><br>    processOrder(order);<br>}</pre><p>Without reading for more than a few seconds, or any comments, we know from the main function that this code validates the order, calculates the total, applies a discount code, and sends a confirmation email. Much easier to read. Much easier to test. Much less likely to have bugs. And let’s face it — is your documentation up to date?</p><h3>Functions Should Return One Type of Thing — Or Nothing.</h3><p>This concept comes from functional coding, and is a generalization of the “Maybe” <a href="https://en.wikipedia.org/wiki/Monad_(functional_programming)">monad</a>, a common structure in functional languages.</p><p>Monads are hard to explain, but the important concept here is that a function shoud not return more than one data type. If it fails, it should return nothing at all, or throw an error.</p><p>The fact that a function should never, ever change the type of its output is something I would call “obvious” but is also why I’m writing this in the first place.</p><p>It’s hard to explain why only returing one type of “something” or nothing is an important practice. In functional programming it allows you a consistent expectation of type, which then lends itself to the blending and chaining of functions on lists of data.</p><p>Outside of strictly functional programming, ensuring there is only one potential output (or nil/null/undefined/None) means there are only 2 code paths that follow from any function. The more code paths, the faster code can turn into spaghetti.</p><h3>Maintain The Current Level of Abstraction</h3><p>You’ve probably worked in many projects that abstract problems into components which are utilized by “orchestration” parts of the system that piece those abstractions together in order to accomplish the actual business logic of your goal.</p><p>Some common examples of this are higher-order-functions, “Models”, and even libraries like ORMs which neatly abstract away the complications of manually writing SQL.</p><p>These systems commonly utilize strategies like <a href="https://en.wikipedia.org/wiki/Dependency_injection">dependency injection</a>/<a href="https://en.wikipedia.org/wiki/Inversion_of_control">inversion of control,</a> <a href="https://en.wikipedia.org/wiki/Currying">currying</a>, <a href="https://en.wikipedia.org/wiki/Higher-order_function">higher-order functions</a> (as previously mentioned), and <a href="https://simple.wikipedia.org/wiki/Class_(programming)">classes</a>. It’s such a common concept that it hardly warrents mentioning. It’s almost part of the very idea of writing code, inseparable from the act itself. Building tools and piecing together systems with your separately conceptualized boxes of “doing stuff” is just <em>how you code</em>. But its worth bringing light to this potentially over-intelectualized concept because this order of compartmentalization is broken <em>all the time </em>and is one of the most common issues with clean code.</p><h4>An Example of Breaking This Principal</h4><p>Here’s the UI for a piece of code I wrote recently in React that renders a staff using <a href="https://opensheetmusicdisplay.org/">OSMD</a> and animates a cursor over the staff line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*YJ2qg_d2J5_Zz2RMSHPhKQ.png" /></figure><p>The actual code for this component looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*zyo-ovp9x7tKzrE91NIGmw.png" /></figure><p>As you can see, all the necessary data for rendering the cursor over the SVG is passed <em>into</em> the cursor component. The cursor is completely self contained and separated from the larger system. That is, until a PR was pushed up to include this code into the cursor component:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_tJ05veP01CF9OwSksvLg.png" /></figure><p>To explain the code a bit (I know it isn’t obvious), this now moves the scroll position of the entire page via a function passed into the component (<em>setScrollPosition</em>) by referencing a height position defined outside the scope of the component (and copied instead of referenced) as well as an index passed in as a prop.</p><p>The comment even references a line from the outer component. All of this should indicate that this logic should possibly be moved to that higher scope.</p><p>Now, we have code in a not-so-obvious place making global changes to the system. I’d hate to be the developer trying to figure out how the scroll is being adjusted, only to find that it’s hidden in the cursor.</p><h3>Everything is a Map-Reduce (Be Declarative)</h3><p>Stealing from functional programming again here. When you get down to it, everything is a <a href="https://en.wikipedia.org/wiki/MapReduce">map-reduce</a>. Once I started using “reduce” in my javascript, I was surprised how many situations it applied to, and similarly with “any” or “all” in python.</p><p>Most problems boil down to applying a similar set of actions to a list of data (map), filtering bad values and/or aggregating the result (reduce).</p><p>By utilizing the libraries and built-in functions that provide these features, your code starts to become a composition of well-understood references instead of a series of step-by-step instructions that need to be read word for word (i.e., more <a href="https://en.wikipedia.org/wiki/Declarative_programming">declarative</a> vs. <a href="https://en.wikipedia.org/wiki/Imperative_programming">imperative</a>).</p><p>It’s a similar conceptually to the levels of abstraction idea mentioned earlier (i.e. higher order functions, inversion of control), but more tactical. You can think of it like telling someone to kneed flower, water, and yeast together to make dough vs using a bread mixer. Functionally, you get the same result, but to kneed the dough by hand, you need to provide additional instructions, like properly applying a layer of flower on the table and on your hands, how to kneed correctly, when to add additional water, and so on. It’s a bit easier to use a tool. Let’s look at an example (again provided by ChatGPT).</p><p>Here’s an imperative example of aggregating a list of records.</p><pre>function calculateTotalSales(sales) {<br>    let total = 0;<br>    for (let i = 0; i &lt; sales.length; i++) {<br>        total += sales[i].amount; // Accumulate the amount sold<br>    }<br>    return total;<br>}<br><br>// Usage<br>const salesRecords = [<br>    { product: &#39;Laptop&#39;, amount: 1000 },<br>    { product: &#39;Phone&#39;, amount: 500 },<br>    { product: &#39;Tablet&#39;, amount: 300 }<br>];<br><br>console.log(calculateTotalSales(salesRecords)); // Output: 1800</pre><p>And here we use a reduce.</p><pre>const calculateTotalSales = (sales) =&gt; <br>    sales.reduce((total, record) =&gt; total + record.amount, 0); // Using reduce to accumulate total sales<br><br>// Usage<br>const salesRecords = [<br>    { product: &#39;Laptop&#39;, amount: 1000 },<br>    { product: &#39;Phone&#39;, amount: 500 },<br>    { product: &#39;Tablet&#39;, amount: 300 }<br>];<br><br>console.log(calculateTotalSales(salesRecords)); // Output: 1800</pre><p>By using the “reduce” syntax, we remove much of the busy work of reading and comprehending a for loop. Here’s a simiar example using python’s “any” function:</p><p>imperative:</p><pre>def has_available_products(products):<br>    for product in products:<br>        if product[&#39;in_stock&#39;]:<br>            return True  # Return true if any product is in stock<br>    return False  # Return false if none are in stock<br><br># Usage<br>products = [<br>    {&#39;name&#39;: &#39;Laptop&#39;, &#39;in_stock&#39;: False},<br>    {&#39;name&#39;: &#39;Phone&#39;, &#39;in_stock&#39;: False},<br>    {&#39;name&#39;: &#39;Tablet&#39;, &#39;in_stock&#39;: True}<br>]<br><br>print(has_available_products(products))  # Output: True</pre><p>declarative:</p><pre>def has_available_products(products):<br>    return any(product[&#39;in_stock&#39;] for product in products)  # Using any to check for in-stock products<br><br># Usage<br>products = [<br>    {&#39;name&#39;: &#39;Laptop&#39;, &#39;in_stock&#39;: False},<br>    {&#39;name&#39;: &#39;Phone&#39;, &#39;in_stock&#39;: False},<br>    {&#39;name&#39;: &#39;Tablet&#39;, &#39;in_stock&#39;: True}<br>]<br><br>print(has_available_products(products))  # Output: True</pre><p>Functionally the same, but over the course of many lines of code, the latter will yield a more readable codebase, and the former will compound into a more tiring experience for the developer.</p><h4>Honorary Mention: Magic Strings and Numbers</h4><p>Don’t <a href="https://en.wikipedia.org/wiki/Magic_number_(programming)">do it</a>.</p><h3>Conclusion</h3><p>If you’re reading this, you may or may not have learned anything new. I imagine that, like me, many practice subconscious principles without much additional thought. Sometimes code just “feels wrong” because of some deeply ingrained habit you don’t even remember learning.</p><p>I’ll check myself the next time I start to think that clean code is “overrated”. But at the same time, when it comes to writing clean code, in my opinion, there are only a few tenants that <em>actually matter</em>. And I wrote this guide as an attempt to name them.</p><p>Once these fairly simple concepts are applied, I dont think there’s actually much else worth discussing. And if you are, you should really just be writing tests instead, which will be the theme of my next post.</p><p>Feel free to argue and tell me I’m wrong in the comments. I probably am.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2be0aecf4b37" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/a-few-unshakable-principles-for-writing-better-code-2be0aecf4b37">A Few Unshakable Principles for Writing Better Code</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing User Data with Custom AWS Pinpoint Events, Kinesis, Lambda, Eventbridge, Glue, and Athena]]></title>
            <link>https://blog.museflow.ai/analyzing-user-data-with-custom-aws-pinpoint-events-kinesis-lambda-eventbridge-glue-and-athena-f84669c0242b?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f84669c0242b</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[aws-athena]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Thu, 24 Aug 2023 03:11:31 GMT</pubDate>
            <atom:updated>2023-12-30T21:59:28.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IUUVqM4C8wPbV6NfVBwS5w.png" /></figure><p><em>Side note: This pipeline was created for use in a project I’ve been working on with a few friends called </em><a href="https://www.museflow.ai/"><em>Museflow.ai</em></a><em>, where we’re trying to combine flow state, gamification, and (eventually) AI to make learning the piano effortless. Feel free to try out the </em><a href="https://beta.museflow.ai/register"><em>prototype</em></a><em>!</em></p><p>If you’re reading this you’re most likely already aware of AWS’s many useful cloud features which make them one of the leading destinations for cloud native applications. AWS’s <a href="https://www.aboutamazon.com/">about page</a> now simply says that they offer “over 200” services, because even they have stopped counting. Today I’ll be focusing on a combination of services which you may or may not have heard of: Pinpoint, Kinesis, Lambda, Eventbridge, Glue, and Athena. That sounds more like a secret pass phrase than a list of technologies, and maybe it is, because together they unlock the ability to analyze your user data as your users interact with your application—or at least I hope.</p><p>My goal in writing this will be to put these services together like lego pieces to arrange a data pipeline that pushed events like logins, sign ups, or really any custom event you choose, to a glue table for you to slice and dice with SQL in Athena. From my understanding of how these services interact with one another, it should very well be possible. I’ll be writing this article in 3 parts:</p><ul><li>Architecture Overview</li><li>Pinpoint setup (with Cognito)</li><li>Front end setup (React)</li><li>Data pipeline</li><li>Athena/Glue</li></ul><p>I’ll be using a combination of two infrastructure frameworks to accomplish my setup: <a href="https://aws.amazon.com/serverless/sam/">AWS SAM</a> and <a href="https://www.terraform.io/">Terraform</a>. I prefer SAM for developing serverless applications for how easy it makes development and deployment, and I like Terraform for shared infrastructure. Configuration values can be shared between these two frameworks using Parameter Store. This project will require both as I’ll be developing a serverless data pipeline in addition to some other infrastructure.</p><h4>Architecture Overview</h4><p>Here’s a quick flow diagram to illustrate the frankenstein of services I will be putting together to achieve this goal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IUUVqM4C8wPbV6NfVBwS5w.png" /></figure><p>To quickly walk through the purpose of each service, pinpoint will be used to collect event data. It offers a convenient way to send events from my user’s application to our backend data lake. It offers a plugin connection to Kinesis that I would otherwise have to manually create. Kinesis similarly plugs directly into Eventbridge. Eventbridge acts as a trigger for my Lambda which will map events to correct S3 buckets. The reason I’ll be using different S3 buckets for different events is to provide a separate schema for each. Glue expects all events in an S3 bucket to hold the same schema in order to partition them into a table, which I will then be able to query with SQL using Athena.</p><h3>AWS Pinpoint Setup</h3><p>Our first order of business is setting up Pinpoint. Since the project I’m setting this up for is a React project, I’ll be showing my frontend client examples in React. The Pinpoint infrastructure setup will be in Terraform.</p><h4><strong>Terraform</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/417454c71e8af3b1255e73ee9c934123/href">https://medium.com/media/417454c71e8af3b1255e73ee9c934123/href</a></iframe><p>This creates an <a href="https://aws.amazon.com/pinpoint/">AWS Pinpoint</a> application and an event stream I can use to send <a href="https://aws.amazon.com/pinpoint/">Pinpoint Events</a>. Not in the above snippet is the role by pinpoint to send events to Kinesis. I create all my IAM roles in a different, global Terraform workspace specific to IAM. I use template files which inherit variables like account id from a global variables file, but here’s the JSON template I use:</p><p>The assume-role policy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2361bc722c65d2ee495cc7bf04d1df05/href">https://medium.com/media/2361bc722c65d2ee495cc7bf04d1df05/href</a></iframe><p>and the policy attachment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/582614575c02cf03dd6110335be20537/href">https://medium.com/media/582614575c02cf03dd6110335be20537/href</a></iframe><p>Now, once I run the Terraform I can see the following in my AWS console when I navigate to <a href="https://us-west-2.console.aws.amazon.com/pinpoint/home?region=us-west-2#/apps">https://us-west-2.console.aws.amazon.com/pinpoint/home?region=us-west-2#/apps</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LU0PalFeBPv3YSqVXm4xgQ.png" /></figure><p>Once you click on the project, the important part is the event stream in settings, which should be enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DbOCQVtv6m4a7u3sE2HBuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AgdV89I4O3wRHXyQ3B1Neg.png" /></figure><p>Now that my event stream is enabled, I can almost start sending events.</p><h4>Cognito Identity Pool</h4><p>Before I can start sending events, my front end needs a way to connect to my AWS resources using AWS credentials. To do that, you need a <a href="https://docs.aws.amazon.com/cognito/latest/developerguide/identity-pools.html">Cognito Identity Pool</a>. Cognito Identity Pools, not to be confused with User Pools, allow one to provide guests and authenticated users with the credentials needed to access AWS resources. They’re provided permissions like anything else in AWS — with an IAM role. However, the assume role permissions is a bit unique. You’ll be specifying a Cognito identity as a federated identity. Here’s what that looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a95d9033186a5fc5ac27f10b78ee30b/href">https://medium.com/media/0a95d9033186a5fc5ac27f10b78ee30b/href</a></iframe><p>As you can see, you’re providing the specific cognito identity with permission to assume the role. This can be with anauthorized access — which would be to send events before a user has logged in — or authorized access, which requires a cognito JWT token.</p><p>You then give your role permission to send events to pinpoint, by attaching a policy that looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d87f087a11854cdb4156e8cdae0a11f/href">https://medium.com/media/1d87f087a11854cdb4156e8cdae0a11f/href</a></iframe><p>This is the part that had me confused for a while: you have to specify both unathenticated and authenticated if you want to be able to send sign up events (because the user hasn’t signed in yet). Also, You will need to attach the IAM role to the cognito identity pool for <em>both</em> authenticated and unauthenticated access. Here’s the Terraform:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5536c9a011c47e14b288189ba4b7e61b/href">https://medium.com/media/5536c9a011c47e14b288189ba4b7e61b/href</a></iframe><p>Something to note about the above. Once your user logs in they will <em>switch </em>from an unauthenticated session to an authenticated session which will change the session ID.</p><h3>React Setup</h3><p>I’ll be using the AWS Amplify project to instrument Pinpoint into my frontend code. They provide several useful Javascript libraries for implementing AWS services. If you haven’t heard of <a href="https://aws.amazon.com/amplify/">Amplify</a>, it’s a framework for developing applications on AWS, similar to <a href="https://firebase.google.com/">Firebase</a>. It’ll get you up and running quickly with AWS infrastructure using Cloudformation. However, I tend to use Terraform for my infrastructure, so I just use the front end libraries they provide.</p><p>However, since they intend people to utilize their libraries with Amplify, they can be a bit cagey in the docs with regard to setting it up without Amplify. All of their docs expect you to import the configuration from a file created by the CLI called “aws-exports”, like so:</p><pre>import { Amplify, Analytics, Auth } from &#39;aws-amplify&#39;;<br>import awsconfig from &#39;./aws-exports&#39;;<br>Amplify.configure(awsconfig);</pre><p>But I don’t really want to do that. Luckily, I was able to find the full expected configuration in one of the library <a href="https://github.com/aws-amplify/amplify-js/blob/main/packages/core/__tests__/parseAWSExports-test.ts">tests</a> which lead me to this config:</p><pre>import { Amplify, Analytics} from &#39;aws-amplify&#39;;<br><br>Amplify.configure({<br>  Auth: {<br>    region: region,<br>    userPoolId: cognitoUserPoolId,<br>    userPoolWebClientId: cognitoClientId,<br>    identityPoolId: awsIdentityPoolId,<br>    identityPoolRegion: region<br>  },<br>  Analytics: {<br>    AWSPinpoint: {<br>      appId: awsPinpointApplicationId,<br>      region: region<br>    },<br>  }<br>});</pre><p>I won’t get into the userPoolId and userPoolWebClientId — that’s for Cognito user authentication and could easily be a second post. With this set up, I can then run the following in my sign up function:</p><pre>import { Analytics } from &#39;aws-amplify&#39;;<br>...<br>return signUp(values)<br>  .then(result =&gt; {<br>    Analytics.record(signUpEvent({email: values.email}))<br>  })<br>  .catch(err =&gt; {<br>    setAuthErrors(handleAuthErrors(err))<br>  });</pre><p>Which uses a defined event function which looks like this:</p><pre>export interface EventAttributes {<br>  [key: string]: string;<br>}<br>export interface EventMetrics {<br>  [key: string]: number;<br>}<br>export interface AnalyticsEvent {<br>  name: string;<br>  attributes?: EventAttributes;<br>  metrics?: EventMetrics;<br>  immediate?: boolean;<br>}<br>export const signUpEvent = ({ email }: {email: string}): AnalyticsEvent =&gt;({<br>  name: &quot;SignUpEvent&quot;,<br>  attributes: {<br>    email<br>  }<br>})</pre><p>Finally, I can see events filtering into the kinesis stream:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1bNT3RNXDcJmsAoFR1Vcg.png" /></figure><p>Now on to the data pipeline.</p><h3>Data Pipeline</h3><p>In order to sent my events from the pinpoint kinesis stream to S3, I’m going to take advantage of <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html">Eventbridge Pipes</a>. Pipes allow you to send events to Eventbridge with additional features like filtering on specific event types and event enrichment. This is the serverless part of the setup, so I’ll be using AWS SAM.</p><p>You can install sam with Pip or any of the options <a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html">here</a>. I’ll be using pip and a python virtual environment. First, I’ll create my virtual env, then I’ll install sam into it.</p><pre>mkvirtualenv pinpoint-event-pipeline<br>pip install aws-sam-cli</pre><p>After a lengthy install I’m ready to run sam init --name pinpoint-event-pipeline . Or, if you already have a SAM repo setup you like, go ahead and copy everything over.</p><h4>Eventbridge Pipes</h4><p>Eventbridge pipes allow you to filter and enrich events on the way to eventbridge from a variety of source types. One of those source types is a kinesis stream, which is why I chose it for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ujjHF0FMnUno_xEpg80VA.png" /></figure><p>The following CloudFormation is what I used to create my pipe:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0baf16817b1bc65456d1ca53f56b800/href">https://medium.com/media/e0baf16817b1bc65456d1ca53f56b800/href</a></iframe><p>Obviously, I cut a lot of the template out for brevity, including the code for deploying my “enrichment” lambda as well as DLQ and Pipeline role. The full template can be found <a href="https://gist.github.com/5t33/8ccfd19b9208fd85ffc0414f38c01e27">here</a>.</p><p>In the above Pipe resource, I’ve defined a filter that only allows events that I’ve defined to pass through to my lambda — specifically only the “SignUpEvent” and “SignInEvent” event types I’ve defined. I’ve also defined an enrichment lambda — which I’ve called a “transformer” lambda because I’ll be using it to transform events into the format I’d like to have sent to my backend. At first, I just used the lambda to print the event to get an idea of how the data is shaped when it gets to my lambda, and without much time passing, I can see it in my Cloudwatch logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wWuo755sXfyQSJKDyqe53g.png" /></figure><p>Unfortunately, at this point I hit a wall. I would like to send the event along its way to Eventbridge (see the target of my Eventbridge Pipe). From there I would be able to create a rule which triggers on arrival of sign in/sign up events. Unfortunately, while I’m able to see the events recieved in the Eventbridge metrics — I’m not able create an event rule that triggers. I’m even using the generated schema from Eventbridge’s handy tool which scans your events:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YdSY7hiDpHFuWDVj32kTiQ.png" /></figure><p>The point of sending the event to event bridge was to create the potential for an event driven approach that would allow me to feed more than one data source from eventbridge. Eventually, if I want to also send data to a realtime database, like <a href="https://www.timescale.com/">Timescale</a>, I would be able to create a second output lambda to route data to that database as well. However, for now my goal is to view my data in Athena. So, instead of routing my event to eventbridge and creating an eventbridge rule to trigger an S3 upload lambda, I’m going to make the lambda the target of my Eventbridge pipe.</p><p>This will involve editing the Pipe config like so:</p><pre>      # Target: !Sub &#39;arn:aws:events:us-west-2:${AWS::AccountId}:event-bus/museflow-events-${Environment}&#39;<br>      # for now - just point to the output lambda instead of pushing to eventbridge first.<br>      Target: !GetAtt PinpointOutputLambda.Arn</pre><p>There are plenty of tutorials on how to set up a lambda with AWS SAM, so I won’t be going over that here. But, there are some gotchyas in the setting up the event pipe and athena backend. You need to make sure your Pipe has permission invoke each lambda (the validator and the final target), and your lambdas need permission to post to the final S3 buckets which will make up your Athena tables. Your lambda invoke permissions will look like this:</p><pre>Resources:<br>  TransformerLambdaInvokePermission:<br>    Type: &#39;AWS::Lambda::Permission&#39;<br>    Properties:<br>      FunctionName: !GetAtt TransformerLambda.Arn<br>      Action: &#39;lambda:InvokeFunction&#39;<br>      Principal: &#39;pipes.amazonaws.com&#39;<br>      SourceAccount: !Ref &#39;AWS::AccountId&#39;<br>      SourceArn: !GetAtt Pipe.Arn<br>  OutputLambdaInvokePermission:<br>    Type: &#39;AWS::Lambda::Permission&#39;<br>    Properties:<br>      FunctionName: !GetAtt PinpointOutputLambda.Arn<br>      Action: &#39;lambda:InvokeFunction&#39;<br>      Principal: &#39;pipes.amazonaws.com&#39;<br>      SourceAccount: !Ref &#39;AWS::AccountId&#39;<br>      SourceArn: !GetAtt Pipe.Arn</pre><p>In addition to the above you’ll want to provide the following permissions to the “output lambda” — or the lambda that your Eventbridge Pipe is targeting:</p><pre>- Effect: Allow<br>  Action:<br>    - s3:PutObject<br>    - s3:AbortMultipartUpload<br>  Resource: <br>    - !Sub &#39;arn:aws:s3:::museflow-sign-up-events-${Environment}&#39;<br>    - !Sub &#39;arn:aws:s3:::museflow-sign-in-events-${Environment}&#39;</pre><h4>Transformation/Validation Lambda</h4><p>Now, I’ll get a bit into the lambda code. My first lambda has two purposes: Validate the incoming event, and transform the event into a truncated version containing only the necessary pieces. The transformer lambda handler looks like this:</p><pre>import urllib.parse<br>import os<br>import traceback<br>from utils.logger import logger<br>import json<br>from aws_xray_sdk.core import patch_all<br>from aws_xray_sdk.core import xray_recorder<br>from main.transformer import transform_event<br><br>logger.info(&#39;Loading function&#39;)<br>env = os.getenv(&#39;Environment&#39;, &#39;local&#39;)<br>TracingOff = os.getenv(&#39;TracingOff&#39;, False)<br><br>def handler(event, context):<br>    log.info(json.dumps(event))<br>    log.info(context)<br>    env = os.getenv(&quot;Environment&quot;)<br>    s3_endpoint_url =  os.getenv(&#39;S3EndpointUrl&#39;)<br>    region = os.getenv(&quot;AWSRegion&quot;)<br><br>    log.info(f&quot;region: {region}&quot;)<br>    log.info(f&quot;s3_endpoint_url: {s3_endpoint_url}&quot;)<br>    validated_events = []<br>    for evt in event:<br>        try:<br>            # print(evt[&#39;data&#39;][&#39;event_type&#39;])<br>            transformed_evt = transform_event(evt)<br>            json_dict = json.loads(transformed_evt.model_dump_json())<br>            validated_events.append(json_dict)<br>        except Exception as e:<br>            if env != &quot;local&quot; and not TracingOff:<br>                subsegment_ref = xray_recorder.current_subsegment()<br>                if subsegment_ref:<br>                    subsegment_ref.add_exception(e)<br>            log.exception(e)<br>            log.error(&#39;failed_to_validate_event&#39;, evt=json.dumps(evt))<br>    try:<br>        return validated_events<br>    except Exception as e:<br>        log.error(e)<br>        traceback.print_exc()<br>        raise e # re-raise for Lambda console</pre><p>Each event is run through a transformation function called “transform_evt”. That function looks like this:</p><pre>def transform_event(event):<br>    if event[&#39;data&#39;][&#39;event_type&#39;] == &#39;SignUpEvent&#39;:<br>        return SignUpEvent(**({<br>            &quot;event_type&quot;: event[&#39;data&#39;][&#39;event_type&#39;],<br>            &quot;datetime&quot;: event[&#39;data&#39;][&#39;endpoint&#39;][&#39;EffectiveDate&#39;],<br>            &quot;session_id&quot;:  event[&#39;data&#39;][&#39;session&#39;][&#39;session_id&#39;],<br>            &quot;email&quot;: event[&#39;data&#39;][&#39;attributes&#39;][&#39;email&#39;],<br>        }))<br>    elif event[&#39;data&#39;][&#39;event_type&#39;] == &#39;SignInEvent&#39;:<br>        return SignInEvent(**({<br>            &quot;event_type&quot;: event[&#39;data&#39;][&#39;event_type&#39;],<br>            &quot;datetime&quot;: event[&#39;data&#39;][&#39;endpoint&#39;][&#39;EffectiveDate&#39;],<br>            &quot;session_id&quot;:  event[&#39;data&#39;][&#39;session&#39;][&#39;session_id&#39;],<br>            &quot;email&quot;: event[&#39;data&#39;][&#39;attributes&#39;][&#39;email&#39;],<br>            &quot;id&quot;: event[&#39;data&#39;][&#39;attributes&#39;][&#39;id&#39;],<br>            &quot;is_test_user&quot;: event[&#39;data&#39;][&#39;attributes&#39;][&#39;is_test_user&#39;],<br>            &quot;user_type&quot;: event[&#39;data&#39;][&#39;attributes&#39;][&#39;user_type&#39;],<br>        }))</pre><p>Each event is being validated by its respective Model, which I’ve written using <a href="https://docs.pydantic.dev/latest/">Pydantic</a>, a rather convenient python validator. Something to point out in the above model is that I’m using the “effective date” portion of the event as the timestamp. It seemed as good an option as any.</p><p>Here’s the model I’m using for my SignUpEvent:</p><pre>from pydantic import BaseModel, Extra, EmailStr, field_serializer<br>from datetime import datetime<br>from typing import Literal<br><br><br>class SignUpEvent(BaseModel):<br>    class Config:<br>        extra = Extra.forbid<br>    @field_serializer(&#39;datetime&#39;)<br>    def serialize_dt(self, dt: datetime, _info):<br>        return dt.strftime(&#39;%Y-%m-%dT%H:%M:%S.%f&#39;)[:-3] + &#39;Z&#39;<br>    event_type: Literal[&#39;SignUpEvent&#39;]<br>    datetime: datetime<br>    session_id: str<br>    email: EmailStr</pre><p>In order for an event to be processed it must conform to the expected schema. Additionally, I’m serializing the date to remove the last 3 0&#39;s that get appended to the end when Pydantic prints the date into JSON. Something else to note is that The Amplify library which I’m using to send these events doesn’t accept null values. So, to compensate for that I wrote the following validator, which allows for nulls:</p><pre>def coerc_string_to_null( string_val: Any):<br>    val = None if string_val == &#39;null&#39; else string_val<br>    return val<br><br>class MyClass(BaseModel):<br>   ...<br>   _my_value_validator = validator(&#39;my_value&#39;, pre=True, allow_reuse=True)(coerc_string_to_null)<br>   my_value: Union[str, None)</pre><h4>“Output” Lambda</h4><p>Now, we can discuss what I’m calling the “OutputLambda”, which is responsible for taking the validated and transformed event, and sending it to the Athena backend. After this section I’m going to go over actually creating the Athena backend, but for now it should be noted that there is an S3 bucket for each event type. Here’s the handler code for the output lambda (I’ve removed some extraneous code that’s pretty much the same as the last one):</p><pre>def get_date_details(datetime_str):<br>    dt = datetime.strptime(datetime_str, &#39;%Y-%m-%dT%H:%M:%S.%fZ&#39;)<br>    return (dt.day, dt.month, dt.year)<br>...<br>try:<br>  s3_client = boto3.client(&#39;s3&#39;, endpoint_url=s3_endpoint_url)<br>  for evt in event:<br>      with tempfile.TemporaryDirectory() as temp_dir:<br>          id=str(uuid.uuid4())<br>          schema = transform_event_schema(evt)<br>          parquet_out(evt, schema, f&#39;{temp_dir}/evt_parquet_{id}.parquet&#39;)<br>          s3_bucket = bucket_map[evt[&#39;event_type&#39;]]<br>          day, month, year = get_date_details(evt[&#39;datetime&#39;])<br>          s3_put_object(s3_client, s3_bucket, f&#39;year={year}/month={month}/day={day}/{evt[&quot;event_type&quot;]}{id}.parquet&#39;, f&#39;{temp_dir}/evt_parquet_{id}.parquet&#39;)<br>  return {<br>      &quot;statusCode&quot;: 200,<br>      &quot;headers&quot;: {<br>          &quot;Content-Type&quot;: &quot;application/json&quot;<br>      },<br>  }</pre><p>As you can see I’m calling a new transform function on each event — but this time it’s called transform_event_schema . That’s because I’m taking each event and creating a parquet schema. You can use JSON for Athena, but Parquet is more efficient, which may save you some $$. Here’s the code for my parquet schema generator:</p><pre>import pyarrow as pa<br>import pyarrow.compute as pc<br><br>def transform_event_schema(event):<br>    if event[&#39;event_type&#39;] == &#39;SignUpEvent&#39;:<br>        return pa.schema([<br>            (&#39;datetime&#39;, pa.timestamp(&#39;s&#39;, tz=&#39;UTC&#39;)),<br>            (&#39;event_type&#39;, pa.string()),<br>            (&#39;session_id&#39;, pa.string()),<br>            (&#39;email&#39;, pa.string())<br>        ])<br>    elif event[&#39;event_type&#39;] == &#39;SignInEvent&#39;:<br>        return pa.schema([<br>            (&#39;datetime&#39;, pa.timestamp(&#39;s&#39;, tz=&#39;UTC&#39;)),<br>            (&#39;event_type&#39;, pa.string()),<br>            (&#39;session_id&#39;, pa.string()),<br>            (&#39;email&#39;, pa.string()),<br>            (&#39;id&#39;, pa.string()),<br>            (&#39;is_test_user&#39;, pa.bool_()),<br>            (&#39;user_type&#39;, pa.string())<br>        ])</pre><p>It’s a bit verbose and frankly, kind of strangely formatted, but that’s the gist. What isn’t pictured here is an example of an int, for which you might use pa.int64() . You can read more about parquet data types <a href="https://arrow.apache.org/docs/python/api/datatypes.html">here</a>.</p><p>Once my parquet schema is created, I can write it to S3. Unfortunately, I couldn’t easily figure out a way to transform the event into parquet and write directly to S3 from memory, so instead I created a file in a temp directory. It’s important to use a temp directory because Lambdas <em>can</em> potentially use the same context with the same temp. The code I used to write the parquet file looks like this:</p><pre>from json2parquet import write_parquet, ingest_data<br><br>def parquet_out(json_blob, schema, path):<br>    # table = pa.Table.from_pydict(json_array, schema)<br>    # pq.write_table(table, path)  # save json/table as parquet<br>    date_format = &quot;%Y-%m-%dT%H:%M:%S.%fZ&quot;<br>    record_batch=ingest_data([json_blob], schema,date_format=date_format)<br>    write_parquet(record_batch, path,compression=&#39;snappy&#39;, use_deprecated_int96_timestamps=True)</pre><p>I used a nifty library called <a href="https://github.com/andrewgross/json2parquet">json2parquet</a>. The one part that’s worth calling out here is use_depreciated_int96_timestamps=True . From the json2parquet docs:</p><p>“If you are using this library to convert JSON data to be read by Spark, Athena, Spectrum or Presto make sure you use use_deprecated_int96_timestamps when writing your Parquet files, otherwise you will see some really screwy dates.”</p><p>Fair enough.</p><p>The final bit of info worth sharing is that when you write these files to S3 you’ll want to do so in binary. Here’s the function I used to read the parquet file and push it to S3:</p><pre>from smart_open import open<br><br>def s3_put_object(client, bucket, key, file_path):<br>    file = open(file_path, &#39;rb&#39;)<br>    content = file.read()<br>    tp = {&#39;min_part_size&#39;: 5 * 1024**2, &#39;client&#39;: client}<br>    uri = f&#39;s3://{bucket}/{key}&#39;<br>    with open(uri, &#39;wb&#39;, transport_params=tp) as fout:<br>        logger.info(f&quot;pushing to s3 {uri}&quot;)<br>        fout.write(content)<br>    file.close()</pre><p><a href="https://github.com/RaRe-Technologies/smart_open">smart_open</a> is utility library which allows you to read and write from S3 like a native file system using the “open” function. As you can see, I’m specifying b for ‘binary’.</p><p>And that’s pretty much it. After deploying, I can go log into my app and watch my function invocation metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZmIZYHMu_Tl0KpKXi5pKg.png" /></figure><p>And check out their corresponding evens in S3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9Y5oKnf-0EYGe3iqkXhTw.png" /></figure><p>You might have also noticed the slightly specific path I chose for these events. This is actually a way that Glue will partition your tables by date automatically (More info in the AWS docs <a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-partitions.html">here</a>), which brings us to our next section: Glue/Athena.</p><h3>Glue/Athena</h3><p>AWS Glue is a pretty nifty tool with a lot of <a href="https://aws.amazon.com/glue/features/">data-centric features</a>. One thing it works well with is <a href="https://aws.amazon.com/athena/">Athena</a> — a Serverless query engine that allows you to query against a multitude of sources, one of which being a Glue table.</p><p>In order to create Glue tables based on your S3 data, you’ll need to create what’s called a “<a href="https://docs.aws.amazon.com/glue/latest/dg/crawler-running.html">Glue Crawler</a>” which will read all of your data in an S3 bucket and place it into a glue table. Now, what’s <em>really</em> nifty about these crawlers is that you don’t even have to create the tables yourself. The crawler will infer the schema of your table based on the format of the data in the S3 bucket. You saw the schemas earlier in my output lambda. So, all you need to do is define an Athena database and one crawler per S3 bucket. Here’s the Terraform I wrote to create them:</p><pre>locals {<br>  glue_role         = data.aws_iam_role.glue_role.arn<br>}<br><br>data &quot;aws_iam_role&quot; &quot;glue_role&quot; {<br>  name = var.glue_role_name<br>}<br><br>resource &quot;aws_glue_catalog_database&quot; &quot;this&quot; {<br>  name         = var.database_name<br>  description  = var.database_description<br>  parameters   = var.parameters<br>}<br><br>resource &quot;aws_glue_crawler&quot; &quot;this&quot; {<br>  for_each = var.tables<br>  name                   = &quot;${var.database_name}_${each.key}_glue_crawler_${var.environment}&quot;<br>  description            = &quot;${var.database_name} glue crawler for table ${each.key} ${var.environment}&quot;<br>  database_name          = aws_glue_catalog_database.this.name<br>  role                   = local.glue_role<br>  schedule               = &quot;cron(0 1 * * ? *)&quot;<br>  configuration = jsonencode(<br>    {<br>      Grouping = {<br>        TableGroupingPolicy = &quot;CombineCompatibleSchemas&quot;<br>      }<br>      CrawlerOutput = {<br>        Partitions = { AddOrUpdateBehavior = &quot;InheritFromTable&quot; }<br>      }<br>      Version = 1<br>    }<br>  )<br>  s3_target {<br>    path = each.value.location<br>  }<br>}<br><br>resource &quot;aws_athena_workgroup&quot; &quot;athena&quot; {<br>  name = var.aws_athena_workgroup_name<br><br>  configuration {<br>    enforce_workgroup_configuration    = true<br>    publish_cloudwatch_metrics_enabled = true<br><br>    result_configuration {<br>      output_location = &quot;s3://${var.s3_query_result_bucket_name}/output/&quot;<br><br>      encryption_configuration {<br>        encryption_option = &quot;SSE_S3&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Not provided above is the config for the S3 Athena query output bucket. Make sure that when you do create the output bucket, you provide a bucket policy that gives access to athena via “athena.amazonaws.com”. For reference, here are the variables I provided to the above template:</p><pre>{<br>  &quot;environment&quot;: &quot;prd&quot;,<br>  &quot;database_name&quot;: &quot;org_name_analytics_prd&quot;,<br>  &quot;database_description&quot;: &quot;org_name athena analytics db&quot;,<br>  &quot;glue_role_name&quot;: &quot;org_name-main-prd-glue-role&quot;,<br>  &quot;s3_query_result_bucket_name&quot;: &quot;org_name-athena-output-prd&quot;,<br>  &quot;aws_athena_workgroup_name&quot;: &quot;org_name-analytics-prd&quot;,<br>  &quot;tables&quot;: {<br>    &quot;org_name_sign_in_events&quot;: {<br>      &quot;description&quot;: &quot;org_name app sign in events table prd&quot;,<br>      &quot;location&quot;: &quot;s3://org_name-sign-in-bucket-prd/&quot;<br>    },<br>    &quot;org_name_sign_up_events&quot;: {<br>      &quot;description&quot;: &quot;org_name app sign in events table prd&quot;,<br>      &quot;location&quot;: &quot;s3://org_name-sign-up-bucket-prd/&quot;<br>    }<br>  },<br>}</pre><p>In this config each key is the name of a table/crawler. Two things are important to mention. Athena only takes underscores and letters as table names, and you need to end your s3 bucket location with a slash: “/”.</p><p>As you can see, I provided a role to be used by the glue crawler. You should make sure that role has permissions to access to each of the S3 buckets you create to hold your parquet events, like so:</p><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;glue:*&quot;,<br>                &quot;lakeformation:*&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::org_name-sign-up-bucket-prd&quot;,<br>                &quot;arn:aws:s3:::org_name-sign-up-bucket-prd/*&quot;,<br>                &quot;arn:aws:s3:::org_name-sign-in-bucket-prd&quot;,<br>                &quot;arn:aws:s3:::org_name-sign-in-bucket-prd/*&quot;<br>            ]<br>        }<br>    ]<br>}</pre><p>Finally, with all this in place I can visit the Athena console and write a query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rSTJ5HkrnZhgZruoSBaUUA.png" /></figure><p>One thing to note is that you should select the correct working group on the top right, because that’s what actually configures the output bucket. Otherwise, you’ll be asked to configure a new one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*4E6_3r_SvkGkG_71SNPKoA.png" /></figure><h3>Conclusion</h3><p>This was a fun project. I’ve wanted to set up an Athena pipeline for a while, but never had a good reason. With an easy to create Athena pipeline, I’ll be able to easily and cheaply store data for use in ML, business analytics, or any other analytics I’d like to do.</p><p>Some closing thoughts. As you saw in the image of my lambda invocation metrics, this process doesn’t batch at all. That means potentially a single invocation per event. This could potentially get rather costly, so a way to mitigate this might be to place an SQS queue between my event pipe and my lambda. Additionally, Crawlers will re-crawl all of your data daily. This can also be costly, but an alternative might be to use <a href="https://aws.amazon.com/about-aws/whats-new/2021/10/aws-glue-crawlers-amazon-s3-notifications/">event driven crawlers</a> which would prevent re-crawling of data.</p><p>Hope this was an enjoyable and instructive read. If you’d like to follow me for more tech articles, feel free to follow me here or add me on Linkedin: <a href="https://www.linkedin.com/in/steven-staley/">https://www.linkedin.com/in/steven-staley/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f84669c0242b" width="1" height="1" alt=""><hr><p><a href="https://blog.museflow.ai/analyzing-user-data-with-custom-aws-pinpoint-events-kinesis-lambda-eventbridge-glue-and-athena-f84669c0242b">Analyzing User Data with Custom AWS Pinpoint Events, Kinesis, Lambda, Eventbridge, Glue, and Athena</a> was originally published in <a href="https://blog.museflow.ai/">The MuseFlow Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[mTLS in 5…10… Okay, 20 Minutes]]></title>
            <link>https://itnext.io/mtls-in-5-10-okay-20-minutes-6602eddae6fe?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6602eddae6fe</guid>
            <category><![CDATA[tls]]></category>
            <category><![CDATA[mtls-authentication]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[ssl]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Sat, 10 Jun 2023 17:39:40 GMT</pubDate>
            <atom:updated>2024-09-01T16:05:22.670Z</atom:updated>
            <content:encoded><![CDATA[<h3>mTLS in 5… 10… Okay 20 Minutes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*ZypLtMHORuG7cFBE_dNiAw.png" /></figure><h3>A Lightbulb Moment</h3><p>I currently work on a team that maintains much of the shared code and networking infrastructure for my company. Considering we’re in the healthcare space, this means a lot of mTLS.</p><p>However, it wasn’t until very recently that mTLS finally “clicked” for me. I learned about it in college and then again later in my career, and then again in my current position.</p><p>Yet, somehow it was still fuzzy. If you’d have asked me 3 months ago if I understood mTLS I would have said “Yes, yes, certificates. They verify us, we verify them, everyone’s verified. Yada yada”. It wasn’t until I had to debug several mTLS configurations in a row in our reverse-proxy that I had an “aha” moment.</p><p>And you probably know how those “Aha” moments go. Suddenly something is illuminated and in no time you can’t even remember why you didn’t understand in the first place. So, I figured I’d take this moment to write down the parts I didn’t understand, what became clear, and how it all works in as ELI5 of a way as possible.</p><p>Edit: Feel free to/please let me know if you find anything incorrect. :)</p><p>Edit 2: After some feedback, I’ve realized that I misconstrued encrypting and signing in a few places and have made updates. I’ve also included some other clarifications where necessary.</p><h3>TLS</h3><p>Before we can talk about mTLS we need to talk about TLS. To understand TLS, you do need a high-level understanding of asymmetric/symmetric key cryptography. Sounds scary, but trust me it’s not that complicated — well, at least all of the complicated parts are abstracted away neatly for us.</p><h4>Symmetric Cryptography</h4><p>Symmetric cryptography is fairly easy to understand. If you’ve ever played around with encoding/decoding ciphers — one of my early memories is doing just that at the Washington, D.C. spy museum, you probably already have an understanding of the gist. Each party has an understanding of the cipher — a process, which allows them to encode/decode text by following an agreed-upon set of steps to encrypt and decrypt. It could be as simple as moving the last letter to the front and the first letter to the back. Using that cipher, you would understand that when I spell my name: “Nteves” all you need to do is reverse the process to get “Steven”.</p><p>Symmetric cryptography works like this. Except, instead of an agreed-upon set of steps, we have an agreed-upon algorithm and secret value. The secret value is what makes the cipher secure. Imagine if, instead of replacing the first and last letter in my name, we had previously agreed upon replacing two random places in my name. The code is harder to crack because it could be any two places.</p><p>Actual symmetric encryption is pretty much the same, but with a much more complicated algorithm, which is <em>extremely</em> hard for an interceptor to crack without knowing the secret value. <strong>In TLS, asymmetric encryption is used to establish this secret value but symmetric encryption is used to encrypt the channel. </strong>Now, let’s talk about asymmetric encryption.</p><h4>Asymmetric Cryptography</h4><p>Whereas Symmetric cryptography is bidirectional, asymetric cryptography is <strong>one way. </strong>If we take the previous example, it’s like if I encoded my message “Nteves” and somehow <em>wasn’t able to change it back</em>. Obviously, I know what’s in the message because I encoded it, but I can’t decode it.</p><p>In Asymmetric cryptography, each party has two secret values which can be used to decode the others’ message. In practice, one is arbitrarily kept “secret” or “private” and the other is “public”. Hence the monikors, “public” and “private keys.</p><p>This allows for two things:</p><ol><li>Anyone can encode a message to the private key holder and know that only they can read it (other copies of the public key can’t be used to decode it).</li><li>The private key holder can encode a message that anyone can decode, thus verifying the message is authentic because it could only have come from the private key holder (remember, it’s kept secret).</li></ol><p>An asymmetric cryptographic algorithm (also called public/private key cryptography) will provide you with a public and a private key. RSA is an example of this algorithm. Commandline tools like <a href="https://www.openssl.org/">openssl</a> are a handy way to utilize these algorithms in practice.</p><p>So, public keys are given out freely. They allow anyone to encrypt something and send that message to the private key holder. But <em>only</em> that private key holder can read the message. This can be used to prove someone’s identity. Just like a password, only one person or entity should have access to the private key, which proves they are who they say they are.</p><p>Here’s a fun little diagram with some arrows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*coAJ-JrYs7YaLjbSNFl7sw.png" /></figure><p>Edit: Some additional points to mention here that have been brought up since publishing.</p><ul><li>This section was intended as an overview of asymmetric cryptography, not only how it pertains to TLS. So, I mentioned the fact that a private key can encrypt a message which a public key can decrypt. <strong>This does not actually happen during the TLS protocol. During an SSL handshake, only the public key is used to encrypt. </strong>In practice, the private key is used to sign data (which I’ll get into) but not encrypt it.</li><li>In RSA Public keys can encrypt messages, but only small ones (sometimes as little as 240 bytes). As such, the public key is only used in the negotiation phase of the handshake to establish symmetric encryption, which is then used for the final encryption of the channel. Later sections explain the application of asymmetric cryptography specific to TLS but it bears mention here to avoid confusion. In practice, other asymmetric algorithms are used for key exchange, like diffie hellman.</li><li>Prior to this edit, the previous section stated that “any public key can decrypt a message encrypted with any private key” and vice versa. The intention of the statement was: “Any copy of a public key can decrypt a message encrypted by its corresponding private key”, not that public keys generated by separate private keys can decrypt each other’s messages. Definitely badly worded.</li></ul><h4>Signing and Digital Signatures</h4><p>As I mentioned above, public/private keys don’t encrypt a lot of data. So, in practice, keys will be used to encrypt a “hash” of the data.</p><p>Hashing is the process of turning data into a unique string of fixed length. Encrypting the hash of data is called a “digital signature” and can be provided alongside the unencrypted data to prove its authenticity. It can be used to verify that the data was sent by the owner of a private key, or at least certified to be true by the owner of the private key.</p><p>In TLS, digital signatures are used both to verify the identity of the website as well as in the negotiation phase of the SSL handshake in which the server sends its supported TLS versions and cipher suites.</p><h4>Certificate Authorities</h4><p>Cryptography is most of what enables TLS. The next part is a Certificate Authority. Certificate Authorities are another clever implementation of asymmetric cryptography. There’s definitely a rectangle-is-a-square thing going on here.</p><p>A public/private key will tell you who I am and allow you to communicate with me securely… but who the hell am I <em>anyway?</em> Sure — I’m Steven, a software engineer that likes to write things online, but why should you care?</p><p>This is where Certificate Authorities come into play. CAs tell you: “this guy is actually <em>google.com</em> so listen up”. When you purchase a web domain, you can request a certificate from a CA that indicates: “I own this domain” (or, more specifically, the entity that holds the private key for this domain owns this domain). This is done by providing a <em>public key</em> to the CA in the form of a CSR (Certificate Signing Request). The Certificate Authority signs (aka hashes, and encrypts) the public key, which grants its authenticity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GlKlvuQsiMFWtweD3s-IQg.jpeg" /></figure><p>Certificate Authorities are institutions set up to verify web domain ownership, which enables encryption online. They’re set up by people that have an interest in maintaining the function of the Internet. Sometimes they’re private companies, like GoDaddy. Other times they’re non-profits, like <a href="https://letsencrypt.org/">letsencrypt</a>. Here’s a list of the top 5 certificate authorities: <a href="https://w3techs.com/technologies/overview/ssl_certificate">https://w3techs.com/technologies/overview/ssl_certificate</a>.</p><p>But, what they really are is a private key. A reaaaally secret one. And a certificate from one of those authorities is really just a document signed with that private key which states: whoever has the specific private key issued for “&lt;insert your domain&gt;.com” is who they say they are. Here’s a diagram that illustrates the process, with a few fibs for simplicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uSHooDPXdYrFHWcCO7v7HA.png" /></figure><p>As I mentioned, the above image has a few lies.</p><ul><li>First, your browser doesn’t actually reach out to a certificate authority for its public key. Your laptop/computer comes with CA public keys for most of the trusted CAs that exist.</li><li>Second, there’s more information sent back and forth, including the TLS version to use, which is left out for simplicity.</li><li>Third, I haven’t included information about intermediate CA’s, also for simplicity. I may add a section about this later.</li></ul><h3>Setting Up a TLS Connection</h3><p>At this point, you should understand the files necessary to create a TLS connection. But let’s list them:</p><ul><li>A public/private keypair</li><li>A CA certificate</li></ul><p>If you’ve been paying attention you may have noticed that the above list is a bit redundant, because a CA certificate includes the public key. But more to the point, how are these files used in practice?</p><p>Let’s look at a simple implementation of TLS on an HA Proxy server (taken from <a href="https://www.haproxy.com/blog/haproxy-ssl-termination">this post</a>)</p><pre>frontend www.mysite.com<br>    bind 10.0.0.3:80<br>    bind 10.0.0.3:443 ssl crt /etc/ssl/certs/mysite.pem<br>    default_backend web_servers</pre><p>I chose HA Proxy because it’s configuration is easy to understand but HA Proxy is a bit weird in that it expects the private key and the certificate in the same file, but know that mysite.pem is actually <em>both</em> a certificate and a private key, and that <em>only</em> the certificate is sent to the client even if they’re in the same file.</p><h4>Curl</h4><p>To send a request to that server, you might use curl:</p><pre>curl https://www.mysite.com </pre><p>Pretty simple now that we know how TLS works, right? But what happens when a site isn’t secure — i.e. the certificate is invalid? Curl will give you an error if you attempt to connect to a website over https. Well, you can turn off certificate validation using -k , like so:</p><pre>curl -k https://www.mysite.com</pre><p>This ^ will become important in a bit, I promise.</p><h3>mTLS</h3><p>So, we’ve set up a TLS connection. Now, how do we make this an <em>m</em>TLS connection? Like this -</p><pre>curl https://www.mysite.com --cert client.crt --key client_private.key</pre><p>What makes this mTLS? Well, as you saw in the previous section, you don’t need to send anything additional to the server to make a TLS connection. The client sends nothing and validates what the server sends back.</p><p>Here, the client (the one initiating the connection) is sending a certificate <em>as well</em> and also specifies a private key. This isn’t necessary for a TLS connection. So, whenever you see a client sending a certificate, you know it’s mTLS.</p><p>Sending a certificate prompts the server to also verify the client’s cert. The server side remains unchanged. However, you might also add a CA certificate, using ca-file , like so:</p><pre>frontend www.mysite.com<br>    bind 10.0.0.3:80<br>    bind 10.0.0.3:443 ssl crt /etc/ssl/certs/mysite.pem ca-file /etc/ssl/ca.crt<br>    default_backend web_servers</pre><p>You might do this because, in many mTLS scenarios, entities use private certificate authorities. This tells HA Proxy <em>how</em> to validate the incoming certificate. The OS of the HA proxy server also might not come with any CA certificates installed because it isn’t necessary for plain TLS.</p><p>But, given that both the client and server are using certificates issued by global, trusted CAs, and that each system has a list of those CA’s certificates installed, there’s really no more that needs to be done.</p><h3>Pop Quiz</h3><p>Given an mTLS connection initiated by curl, what happens when -k is used, like so:</p><pre>curl -k https://www.mysite.com --cert client.crt --key client_private.key</pre><p>Which sides of the request, if any, are still validated? I’ll include the answer in a comment.</p><h3>Conclusion</h3><p>I hope reading this article was able to make mTLS a bit clearer for you, as writing it has for me. I’d like to thank all of those who gave constructive feedback as it was very helpful in patching up some of the mistakes in the article and helping in my own understanding.</p><p>In the end, the mTLS section was much shorter than the lead-up, because what’s hard to understand about mTLS is the TLS part. Once you understand TLS, mTLS is just… TLS again.</p><p>Happy networking, and feel free to follow me on Linkedin.</p><p><a href="https://www.linkedin.com/in/steven-staley/">https://www.linkedin.com/in/steven-staley/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6602eddae6fe" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/mtls-in-5-10-okay-20-minutes-6602eddae6fe">mTLS in 5…10… Okay, 20 Minutes</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Continuous Infrastructure Deployment with Terraform Cloud]]></title>
            <link>https://itnext.io/continuous-infrastructure-deployment-with-terraform-cloud-2853cbbb920?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2853cbbb920</guid>
            <category><![CDATA[sentinel]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[terraform-cloud]]></category>
            <category><![CDATA[hashicorp]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Thu, 27 May 2021 23:55:41 GMT</pubDate>
            <atom:updated>2022-06-26T05:47:36.725Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*etej6CTBPY8VLaGLGiIhQw.png" /></figure><h3>Terraform &amp; Automation</h3><p>At my last company, we used Terraform a lot. It allowed us to create infrastructure repeatably and leverage each others’ code to do so. It also gave us a standard method to make infrastructure changes without teams writing their own ad-hoc tooling, all while documenting that infrastructure in the form of code! If you’re familiar with Terraform, or other infrastructure as code tools you probably already understand and reap the benefits of using them as well.</p><p>However, there comes a point where Terraform requires shared structure and workflows in order to keep environments consistent and enable collaboration among team members. For this, you might use remote state, workspaces, or tools like <a href="https://terragrunt.gruntwork.io/">Terragrunt</a>. This is also where options like <a href="https://www.terraform.io/docs/cloud/index.html">Terraform Cloud</a>, <a href="https://spacelift.io/">Spacelift</a>, <a href="https://www.env0.com/">env0</a>, <a href="https://scalr.com/">Scalr</a>, or <a href="https://www.runatlantis.io/">Atlantis</a> come into play. These Terraform automation tools <a href="https://www.youtube.com/watch?v=4MLBpBqZmpM">coined “TACOS” by CloudPosse</a> (Terraform Automation &amp; Collaboration Software) provide a variety of benefits that are hard to match when rolling your own infrastructure automation. These include:</p><ul><li>Hands off, secure backends for your Terraform state</li><li>Shared plan output for easier collaboration</li><li>Terraform Version Locking</li><li>Additional permission management outside of version control</li><li>Webhooks and “triggers” of one workspace based on the run of another</li><li>Policy as Code frameworks like <a href="https://www.openpolicyagent.org/">OPA</a> or <a href="https://www.hashicorp.com/sentinel">Sentinel</a></li><li>API driven Terraform runs</li><li>Cost estimation</li><li>State auditing &amp; diffs</li><li>A private module registry</li></ul><p>While these tools tend to have a similar set of features, today I will be exploring <a href="https://www.terraform.io/docs/cloud/index.html">Terraform Cloud</a> to automate infrastructure changes via GitHub Actions and promote those changes from one environment to the next. As a demonstration, I’ll be creating a small AWS module with 3 different environments: “tst”, “stg”, and “prd”. Each of these environments will be in the AWS region “us-west-2”.</p><p>The goal of this post will be to explore 3 things:</p><ol><li>Handling multiple environments in the same module using “workspaces” and the Terraform Cloud backend.</li><li>Automating the process of applying changes from one environment to the next.</li><li>Using Sentinel policies which prevent infrastructure updates that don’t conform to spec.</li></ol><h4>Update 06/02/21</h4><p>I was informed after posting this article that it actually is possible to use the -tfvars option in Terraform Cloud by providing the TF_CLI_ARGS environment variable. While I haven’t tried it yet, this would solve the main issue I had with Terraform Cloud while creating this demo, which as you’ll see was not being able to provide a specific *.tfvars file for each specific environment, meaning I had to do some code adjustment to compensate. Instead, I could have set TF_CLI_ARGS in each workspace to the specified tfvars . If that doesn’t make sense it should by the end of the post.</p><p>Shout out to <a href="https://medium.com/@jeremygaither">Jeremy Gaither</a> for pointing that out.</p><h3>Project Setup</h3><p>Without wasting any more time, let’s get into the code (which you can find <a href="https://github.com/5t33/tf-cloud-workspaces-demo">in this repo</a>).</p><p>The folder structure I’ll be using will be as follows:</p><pre>├── module<br>├── sentinel<br>├── usr_mgmt<br>└── workspaces</pre><p>If this were a real environment each of these folders represents a set of resources that might be its own repository, but for the purpose of this demo I’ll be keeping them together. Without worrying too much about the other folders for now, the resources I will be deploying will go into the module directory.</p><p>Here’s the simple module I’ll be deploying:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a97f659ba888877e2e66c5e8472825fc/href">https://medium.com/media/a97f659ba888877e2e66c5e8472825fc/href</a></iframe><p>And the variables accompanying it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e426c19a30159319df05b38876b3a170/href">https://medium.com/media/e426c19a30159319df05b38876b3a170/href</a></iframe><p>(edit: This is where the 06/02/21 update would change things. Instead of one vars file, each of these blocks would be in its own environment-specific vars file, e.g. prd-us-west-2.json . That vars file would then be specified in the automation process via the TF_CLI_ARGS environment variable, e.g. TF_CLI_ARGS=&quot;-var-file=prd-us-west-2.json&quot; )</p><p>You might immediately notice some odd things about my variable declarations. Instead of defining each variable on its own, I’ve defined them all inside of a map called “variables”. This isn’t a normal pattern.</p><p>If you’re familiar with Terraform, you know that generally different input variables are provided via variable files or variable arguments. For each environment, a specific set of config variables can be provided in a file using the -var-fileflag or via -var. However, Terraform Cloud doesn’t currently support providing vars or var files via the CLI.</p><p>Instead, you have three choices:</p><ol><li>Use the <a href="https://registry.terraform.io/providers/hashicorp/tfe/latest/docs/resources/variable">Terraform Cloud variable resource</a> for all of your variables (which is extremely cumbersome and completely unrealistic).</li><li>Use completely different folders with the same code, or potentially a call to a module.</li><li>Make each of your variables a map and selecting the variable depending on the workspace, which is what I’m doing here.</li></ol><p>My workspace is defined by a combination of the environment( tst, stg, or prd) and aws_region. These are then concatenated as a local variable inside the module, which gives me tst-us-west-2, stg-us-west-2, and prd-us-west-2. As you’ll see in a second, these two variables (environment and aws_region) will be supplied to the workspace via Terraform cloud, so the developer wont have to worry about setting them. They can then be used to retrieved environment specific variables, like so:</p><pre>var.variables[local.workspace].&lt;variable name&gt;</pre><p>To be honest, I really don’t <em>love</em> having to change my code in order to accommodate different an automation solution. It feels like sacrificing my module’s logic for CI. I’ve generally gone with the second option listed above, however workspaces are <em>supposed</em> to give you the option to deploy different environments from the same module in order to keep your code dry, and I wanted to give it a shot. Hopefully Hashicorp adds -var-file as an option soon, because not having it is kind of a drag.</p><h3>Workspaces</h3><p>Before I can deploy this resource using Terraform Cloud, I’ll have to set up my workspaces. If you’re following along and you don’t yet have a TF Cloud account, you can sign up for one <a href="https://app.terraform.io/signup/account">here</a>. Quick aside on <a href="https://www.terraform.io/docs/cloud/workspaces/index.html">workspaces</a>. A workspace is a Terraform concept where multiple states can exist for the same module, commonly used to differentiate between environments, e.g. test, staging, production. A “workspace” maps 1:1 with a state file.</p><p>Confusingly, traditional CLI workspaces are not the same as Terraform Cloud “workspaces” as you can see from this snippet of the docs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drsdPQ_-QnhhakKIpXtV7g.png" /></figure><p>Terraform Cloud workspaces are described like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKe3OIed3aGa-8VhDh8tgg.png" /></figure><p>I get what they’re saying… but in the end there is still one state for each workspace and they work interchangeably, so I don’t <em>really</em> understand why its implied that Terraform Cloud workspaces aren’t alternate state files. I suppose it’s because In TF Cloud a “workspace” also means the product features associated with it, but it’s oddly worded. Anyway, to understand how workspaces… work, it’s good to set one up in the TF Cloud console first, which gives you the following setup screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*gTR3mv6mzT9mygYcuAJvVQ.png" /></figure><p>There are three options to choose from. Each has its pros and cons. Let’s examine them.</p><h4>VCS Workspaces</h4><p>VCS workspaces are by far the easiest to set up. It handles a lot of automation for you. Without needing to add your own build step, Terraform Cloud will clone your repository, run a plan, and provide a nice green check once the plan passes (and what developer doesn’t like green check marks?):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4uuumXEj0bx53doL9WMzjA.png" /></figure><p>If your repository has submodules, it handles this too granted a satisfactory ssh key is provided. Once the PR is merged it will begin an apply run which must be approved in the Terraform Cloud console (if required). There are a lot of pros to this option. It’s probably the easiest way to get set up with TF Cloud Automation and it prevents applies from the console, which you might want to restrict. However, it’s a bit limiting as well and there <em>are</em> some cons.</p><h4>VCS Cons</h4><p>As mentioned, you won’t be able to run applies from the command-line, which you might want, but it also pretty limiting. TF Cloud <em>doesn’t</em> give you the option to run additional code alongside your plan/apply steps in a build script (some of the other TACOS do) so you might want to opt for the more manual CLI approach to give you that flexibility.</p><p>Additionally, if you only work off of a single main branch in version control, you won’t really be able to promote changes from one environment to the next on your own terms. All applies are queued at once when the PR is merged. This isn’t really a problem for a repository with a multiple branch git-flow setup, e.g. “develop”, “staging”, and “main”. However, if you rely on your CI/CD pipeline to roll out changes to each environment in succession, it’s not so great. With one main branch and multiple workspaces, order is not guaranteed. Runs happen in any order that you approve them in the TF Cloud Console. Let’s say you merge and the test and staging environments start a run and are waiting for you to hit “approve” on each to initiate an apply. You could feasibly lick “run: the staging environment before test, which shouldn’t be allowed. Additionally, considering that all runs are queued and TF Cloud has concurrency limits, it’s also possible that you’d be forced to apply production before test because test is stuck in the queue!</p><p>Also, because VCS workspaces only include a link to the plan result in the PR, all users must have an account to view the plan. If you’re running your plan more manually you can make the plan output to github in a comment just part of the flow. For these reasons, I’m going to opt for CLI workspaces for the rest of the demo (also, VCS workspaces are fairly easy to figure out anyway).</p><h4>CLI Workspaces</h4><p>CLI workspaces are a bit more manual in that you have to run a plan and apply yourself in your CI environment, but gives you more control over when the apply happens, as opposed to happening immediately upon merge. In the case where you only have one main branch, you can build a deployment pipeline that sequentially applies changes to each of your workspaces. Also, you can add additional steps, like smoke tests after your apply script. With the VCS option you’re stuck with a webhook to add additional automation and checks after your apply.</p><p>However, the setup is more complicated and you’ll need to add a TF Cloud Token as secret for Github Actions (or wherever CI tool you use to run your Terraform).</p><h4>API Workspaces</h4><p>API workspaces aren’t <em>really</em> a separate type of workspace. You can apply them from the commandline like CLI workspace, and the /run API endpoint can be used on CLI, API, or VCS workspaces. The setup for a CLI and an API workspace is exactly the same as far as I can tell. I think TF Cloud just included it as an option in the builder to point out that running your Terraform with an API call is a feature. For the record, creating an apply run with a workspace looks like this:</p><pre>export TOKEN=&quot;&lt;your token&gt;&quot;<br>curl \<br>  --header &quot;Authorization: Bearer $TOKEN&quot; \<br>  --header &quot;Content-Type: application/vnd.api+json&quot; \<br>  --request POST \<br>  --data @create_run.json \<br>  https://app.terraform.io/api/v2/runs</pre><p>where your data is:</p><pre>{<br>  &quot;data&quot;: {<br>    &quot;attributes&quot;: {<br>      &quot;is-destroy&quot;: false,<br>      &quot;message&quot;: &quot;This is an API driven run&quot;<br>    },<br>    &quot;type&quot;:&quot;runs&quot;,<br>    &quot;relationships&quot;: {<br>      &quot;workspace&quot;: {<br>        &quot;data&quot;: {<br>          &quot;type&quot;: &quot;workspaces&quot;,<br>          &quot;id&quot;: &quot;id-xxxxxxxxx&quot;<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>However, with only the above initial setup, this would only work off the bat for a VCS workspace. In order to do a remote run, TF Cloud needs the terraform resource files to run and compare against state. One of the advantages of a VCS workspace is that on each run TF Cloud will pull files from your version control provider. A CLI workspace will require you to <a href="https://www.terraform.io/docs/cloud/run/api.html">upload a “configuration” in the form of a zip file</a> with your Terraform code.</p><h3>Terraforming Terraform</h3><p>It’s fairly simple to click around in the TF Cloud console to set up your workspaces, but this isn’t very repeatable so I made this small Terraform module to do it. “Terraforming Terraform” like this is actually one of the reasons Terraform Cloud is a nice option, but other TF automation solutions offer providers as well.</p><p>Here’s an example of a workspace resource using the Terraform Cloud provider:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/662306f57169b9c6a84d06569c72eb90/href">https://medium.com/media/662306f57169b9c6a84d06569c72eb90/href</a></iframe><p>And here is the generalized module I created (by the way this lives in the workspaces section of the demo repository):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7645f5e95ac8a84563c69d06ce52cad/href">https://medium.com/media/b7645f5e95ac8a84563c69d06ce52cad/href</a></iframe><p>As you can see it also reaches out to a child module called “tf-cloud-variables”, a module I created to create Terraform Cloud variables with the option to pull sensitive data from AWS SSM Parameter store. That module looks a bit like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60a86793026c03ecfca68a37c48b194a/href">https://medium.com/media/60a86793026c03ecfca68a37c48b194a/href</a></iframe><p>Of course, it has a bunch of stuff removed. You can see the full module in the repo. The point is to allow me to include my variable &amp; environment variable declarations alongside my workspace declaration as you can see in the example tfvars file below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8b6f2fbc9d31e1d8a8d7f8cc47ec0df/href">https://medium.com/media/c8b6f2fbc9d31e1d8a8d7f8cc47ec0df/href</a></iframe><p>I removed the other two workspaces to cut down on space but they’re exactly the same with the exception of environment values/workspace names. By setting vcs_repo to null the workspace becomes a CLI workspace. There’s no way to choose CLI or API (which makes me think they’re kind of the same thing). As you can see I’ve included two variables here: environmentandregion that will be used in the previous module to create the local variable: workspace, which will allow the module to pull the correct config for the environment in which it’s being run.</p><p>However, before the workspaces can be set up, the Terraform Cloud provider needs to be added:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1ead32d2d892670a17464dcdff3df13/href">https://medium.com/media/e1ead32d2d892670a17464dcdff3df13/href</a></iframe><p>The tfe_token will need to be a <a href="https://www.terraform.io/docs/cloud/users-teams-organizations/teams.html">team</a> or <a href="https://www.terraform.io/docs/cloud/users-teams-organizations/users.html">user</a> token which can be created on the TF Cloud app. I added this variable to a secrets file included in my gitignore called secrets.auto.tfvars. With this in place using the correct variables a workspace will be created for each environment of the module.</p><p>After applying the workspaces module you should see them created in the TF Cloud console (and in your terraform command-line output):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4DoG8vjVzovmdLnR2wghg.png" /></figure><p>(the “applied” bit wouldn’t be there yet, but I took this picture a while after creating the initial workspaces).</p><p>By the way, I didn’t set up a backend for this workspaces module, but ironically you would probably have to do so in the console initially, or create the workspace module’s workspace without a workspace and then add it after the fact… meta.</p><h3>Terraform Cloud Backend</h3><p>Now that the workspaces have been set up we can actually start using them! To do so we’re going to head back over to the S3 module and add the Terraform Cloud Backend, which looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ac7c86cf49601a92e9ba66c3e560c87/href">https://medium.com/media/9ac7c86cf49601a92e9ba66c3e560c87/href</a></iframe><p>This will replace whatever backend you’re currently using. One thing to note is the prefix setting. This is how Terraform Cloud sets up multiple workspaces for a single module. Any workspace beginning with the above prefix tf-cloud-workspaces-demo- will be an option when you run terraform init .</p><p>However, before you can run plan you also need to set up your .terraformrc config with your token. This can be accomplished by running terraform login or by copying the following into a file in your home directory called .terraformrc</p><pre>credentials &quot;app.terraform.io&quot; {<br>  token = &quot;&lt;your token here&gt;&quot;<br>}</pre><p>Following that, when you run terraform inityou will be greeted with the following input:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNIJNqbK1LUppILAJK5tqQ.png" /></figure><p>From there you can choose the workspace you would like to use locally to run plans.</p><h4>Local Workspaces</h4><p>Workspaces are meant to be run in a consistent environment on TF Cloud servers (this might be another reason they like to differentiate from “CLI” workspaces). So even when you run your plans locally, they’re actually streaming results from a TF Cloud server. However, if you want to run the Terraform on your own computer and <em>just</em> use TF Cloud for the state management, you can do so by setting your workspace to “local”.</p><h3>IAM &amp; Assume Role Provider</h3><p>There’s one more necessary step before these resources can be deployed. I will be running this Terraform using a roll since this will be automated and I won’t be using my own credentials. Because I won’t be using an on-prem agent with Terraform Cloud, I need to provide some separate credentials. This means creating a user with perms to create/update the specified bucket and bucket object. Or, more precisely, a role with permission to create/update the bucket which the Terraform user is able to assume. Since I’m lazy and don’t feel like working for least perms, the policy resource for that assumed role looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e95654f01f5e551f4577ecd0c29846d4/href">https://medium.com/media/e95654f01f5e551f4577ecd0c29846d4/href</a></iframe><p>The for_each here iterates through each workspace, which I mentioned before are tst-us-west-2 , stg-us-west-2 , and prd-us-west-2 . The role for that policy looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b01623ba710fd5bb5f6378ebd3fd7c3c/href">https://medium.com/media/b01623ba710fd5bb5f6378ebd3fd7c3c/href</a></iframe><p>As indicated by this line:</p><pre>AWS = data.aws_iam_user.deployment_user.arn</pre><p>The role is only able to be assumed by the user defined in the console. Now that both the role and the user are created, the module’s AWS provider can be updated like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9d4c9c44dd465e824a69f98514682d7/href">https://medium.com/media/d9d4c9c44dd465e824a69f98514682d7/href</a></iframe><p>As you saw above I provided the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in each of the Terraform Cloud workspaces (using the keys of the user I created earlier with perms to assume the deployment role).</p><p>What’s kind of cool about this is the person running the Terraform doesn’t even need AWS permissions to run plans/applies — they only need permission to access the environment in Terraform Cloud. This significantly reduces the amount of perms lying around with access to change infrastructure.</p><p>Another option here would be to provide temporary credentials for the assumed role via environment variables AWS_ACCESS_KEY_ID , AWS_SECRET_ACCESS_KEY , and SESSION_TOKEN . Then permanent credentials wouldn’t be necessary and the “assume_role” could be removed from the aws provider. However, that would also require some automated process to update the temporary credentials on a daily basis, which I won’t be doing here.</p><p>Now that the provider is in place, I can test it out by running a plan locally, which starts off like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZ2nPDQwAQvBdYngMJMoTA.png" /></figure><p>A quick scan shows that the plan is being run on a linux machine using Terraform v0.15 and a nifty link is provided that can be used to share the output of the plan. Following that we get our plan as well as a cost estimation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*wuZmdvyFKZphsAy0bcWvxg.png" /></figure><h3>Github Actions</h3><p>Now all the pieces should be in place for a fully automated deployment. To get started with Github Actions, you’ll need to add a TF Cloud token to the repository’s secrets. You can find by the secrets section of Github by navigating to <strong>settings</strong> -&gt; <strong>secrets </strong>in your repository. I’ll be naming this token TF_CLOUD_TOKEN .</p><h4>Plans</h4><p>When PRs are created we’ll want an action to run a plan for each workspace. The Github Actions workflow for that looks like this (and as I mentioned before I used <a href="https://learn.hashicorp.com/tutorials/terraform/github-actions">this tutorial</a> for this part of the demo):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d730b2c439e03367f5a71f101ab9ac78/href">https://medium.com/media/d730b2c439e03367f5a71f101ab9ac78/href</a></iframe><p>As before, I removed the stg and prd jobs to save space because they’re exactly the same. To clarify what’s going on here, the on section declares that the following list of jobs will be triggered on pull requests against the main branch. paths indicated which folder paths will be watched for changes. Another important bit is the env section with one environment variable defined: TF_WORKSPACE . This tells Terraform what workspace to pick. Without this, init will require manual input.</p><p>Following this, the job plan-tst performs the following actions:</p><ul><li>Checkout (checkout the repo)</li><li>Setup Terraform (download the required Terraform version)</li><li>Terraform Init (initialize the workspace)</li><li>Terraform Format</li><li>Terraform plan</li><li>Update Pull Request (Comment the plan onto the pull request)</li><li>Terraform Plan Status (Exit 1 if the plan fails)</li></ul><p>These steps are fairly straight forward but there are a few things to highlight. First, cli_config_credentials_token:${{ secrets.TF_CLOUD_TOKEN}} provides the token that I added as a secret to the Github repo. Second, the environment variable TF_WORKSPACE which is set at the job level sets the workspace to tst-us-west-2 . This tells Terraform which workspace to use, otherwise init will require input. Finally, continue_on_error allows the job to continue even when a plan or fmt command fails so that these results can be used as output to the comment. Also, I have the job filtered on the module directory so that only changes to the module will kick off these plans.</p><p>Now, once a PR is created, these plans will run and post their results to the PR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*6khsHZWqIzI2SFeIf4JLRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnc0DwdHSlvRa4Bo4JnBVA.png" /></figure><h4>Applies</h4><p>The GH action above provides a blueprint that can be easily adjusted to run applies in each workspace successively, with approvals between actions. When Terraform Apply is run on a TF Cloud workspace, you have the option to approve it from the TF Cloud console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2l2HLkO8NmJzabmIShu_A.png" /></figure><p>So, there isn’t really any need to add additional logic to add approval gates between environments. Additionally, since Terraform Cloud has its own access-control, so you aren’t dependent on Github authorization for approvals. This is helpful if your Github controls aren’t as tight as you would like for approving infrastructure changes.</p><p>To update the Github Action for applies, I’ll first be changing the on condition to push which means the workflow will be kicked off when the PR is merged. Second I will be adding the need configuration to the stg and prd jobs so that they wait on each other to complete before starting. If one fails, the next will not start. Finally, I’ll be removing the fmt command and the PR comment because they aren’t necessary for the apply stage. In the end we’ll get something like this (this time I left all workspaces in):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7f0100466b5248c7a51a02b31d0f208/href">https://medium.com/media/c7f0100466b5248c7a51a02b31d0f208/href</a></iframe><p>Now when I merge a PR, the job kicks off:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*S-tYPQmi2GjUC0H3kI4NuQ.png" /></figure><p>Which I can then approve in the TF Cloud console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCKnEjP9Z2lOAEgl-C_F2g.png" /></figure><p>And after applying that change, I see the next job start:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*_JO8J7vXAXsn6QfB2Kha-A.png" /></figure><h4>Audit Trails</h4><p>Another cool aspect of Terraform Cloud (and other TACOS) is that state diffs are recorded for every infrastructure update, so after the apply you can go back and see how your state was changed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNu2tzfDhg1BupVltJETRg.png" /></figure><h3>Sentinel</h3><p>Sentinel is Hashicorp’s Policy As Code language which comes built in to a few of their products. With Terraform Cloud, Sentinel allows you to specify rules which plans must adhere to in order to pass. You can enforce restrictions on pretty much anything you want: EC2 size, public S3 buckets, open security groups, wide IAM policies, etc. In fact, since TF Cloud also does cost estimation, you can enforce cost restrictions as well. These enforcements can either be ‘advisory’, ‘hard-mandatory’, or ‘soft-mandatory’.</p><p>As with Workspaces, the Terraform Cloud provider allows you to create Sentinel policies using Terraform. That resource looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cd324e0241accfadbe777ab98799c9e/href">https://medium.com/media/7cd324e0241accfadbe777ab98799c9e/href</a></iframe><p>These policies can be applied to individual VCS repositories, specific workspaces, or globally across all workspaces. I’ll be creating one that will be global policy to ensure all buckets are private, which looks like this:</p><p>must_not_be_public.sentinel</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b82440bd0380b423a29157112649a02/href">https://medium.com/media/8b82440bd0380b423a29157112649a02/href</a></iframe><p>This policy, which is applied to a plan output via the tfplan/v2 import, filters through all resource changes for actions that are “create” or “update” on resources which are “aws_s3_bucket”. It them checks the acl attribute for public options. Then, that same process is done for bucket objects. At the end, each of these lists is checked for a length of 0. This occurs in the main rule, which is the entry point of the policy, (by the way this policy originally came from a Hashicorp demo, but for the life of me I can’t find it).</p><h4>Testing Sentinel</h4><p>To know whether this policy actually does what we want it to, it needs to be tested, which requires mock data. You can download data from any plan run in Terraform cloud by navigating to the run and hitting the “download sentinel mocks” button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKy9S96Ctf6ng_-JyP-oAg.png" /></figure><p><strong>Note: This is called “mock” data but seems to include sensitive values.</strong> <strong>Make sure to scan them before adding to your version control.</strong></p><p>Once these are downloaded, you can unzip them with tar:</p><pre>tar xzf ~/Downloads/run-xxxx-sentinel-mocks</pre><p>A variety of files are provided, one of which will be called mock-tfplan-v2.sentinel . This corresponds to the data available when the tfplan/v2 is used. This is what I’m using to mock my plan for my sentinel tests. One thing to be careful of here is that sensitive data is included in these mock plans, which you’ll want to strip before making a commit.</p><p>Sentinel uses file structure to determine where tests are located. Your sentinel policies should be at the base of your directory and all tests should be in the &quot;test” folder like so: test/&lt;your_policy_name&gt;/*.json . Each sentinel policy has its own folder with potentially multiple tests. Following this format I have the following file structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*OH8ET8k1WgkJD0xqLIF22w.png" /></figure><p>The global folder holds sentinel policies applied globally. If I were to include workspace or repo specific policies I would include them in workspaces/&lt;workspace name&gt; or repository/&lt;repository name&gt; folders. module holds shared Terraform modules that will be shared across multiple folders.</p><p>Each test in file looks something like this:</p><p>fail-bucket.json :</p><pre>{<br>  &quot;mock&quot;: {<br>    &quot;tfplan/v2&quot;: &quot;../../test-data/mock-plan-fail-bucket.sentinel&quot;<br>  },<br>  &quot;test&quot;: {<br>    &quot;main&quot;: false<br>  }<br>}</pre><p>the mock section points to the mock data that was altered to fail the sentinel check for a public bucket, and the test section asserts that the main check is false for the provided mock data.</p><p>After running sentinel test I get the following output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*cSOffhL4PGdv6hLP3lWnPg.png" /></figure><p>All three tests I wrote pass, so I can move on to deployment.</p><h4>Terraforming Sentinel</h4><p>As with the workspaces module from earlier, I also created a sentinel module to centralize and reuse my sentinel code. A sentinel policy Terraform resource looks like this:</p><pre>resource &quot;tfe_sentinel_policy&quot; &quot;test&quot; {<br>  name         = &quot;my-policy-name&quot;<br>  description  = &quot;This policy always passes&quot;<br>  organization = &quot;my-org-name&quot;<br>  policy       = &quot;main = rule { true }&quot;<br>  enforce_mode = &quot;hard-mandatory&quot;<br>}</pre><p>These policies are added to policy “sets” like this:</p><pre>resource &quot;tfe_policy_set&quot; &quot;test&quot; {<br>  name          = &quot;my-policy-set&quot;<br>  description   = &quot;A brand new policy set&quot;<br>  organization  = &quot;my-org-name&quot;<br>  policy_ids    = [tfe_sentinel_policy.test.id]<br>  workspace_ids = [tfe_workspace.test.id]<br>}</pre><p>A policy can either<em> </em>be applied to<em> </em>a set of workspaces or globally across all workspaces using the global flag (in which case workspace ids aren’t provided). As with the other Terraform Cloud provider resources, I’ve bundles these two up into a module that I won’t get into this time, but which allows me to declare my policy sets as a map of workspace names to policies like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/716c88350ebea3a129309cc080deb949/href">https://medium.com/media/716c88350ebea3a129309cc080deb949/href</a></iframe><p>This results in the following resources being created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kcNak1adTplGLDHxjP8yuQ.png" /></figure><p>Now I can test the policy by pushing a change that would make my bucket public. Let’s see what happens when I set acl to “public-read” on my s3 resource like this:</p><pre>resource &quot;aws_s3_bucket&quot; &quot;demo&quot; {<br>  bucket = &quot;tf-cloud-workspaces-demo-${local.workspace}&quot;<br>  force_destroy = var.variables[local.workspace].force_destroy<br>  tags =  var.variables[local.workspace].tags<br>  acl = &quot;public-read&quot;<br>}</pre><p>With the public-read option on, I suddenly get this output when I run a plan (in two photos because my screen wasn’t bit enough):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*t2WtpTWXWRcDXF8T_CJh0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*tzVYnm-Gol5fUzCnM2w2SA.png" /></figure><p>And it failed the check! Although the “empty policy set name” is odd… but it doesn’t seem like a huge deal. Just for the heck of it I’ll also push this to a PR in Github to see the plan fail there as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xXT7MygcqZDCMKXSx4PGJw.png" /></figure><p>And just like in the local run, all my Github checks have failed. Success! Or.. err.. Failure… I guess.</p><h3>Conclusion</h3><p>As you can see… TF Cloud! Nice. It does the things.</p><p>I can see these tools being quite useful for larger organizations so that developers can share runs easily, have a consistent deployment environment, enforce infrastructure policy, monitor costs, provide state audit trails, and potentially run automated infrastructure changes via API calls. However, even for a smaller org, just having Terraform Cloud manage your remote state with locks instead of an S3 bucket and a dynamodb table (or however you choose to do so) is nice. S3 has been the cause of a few high profile security incidents, so shipping that responsibility to a managed service can be helpful. And you can always add on features like remote execution &amp; sentinel policies later.</p><p>That being said, some of the other services I listed in the beginning also have strong offerings and may beat the price of Hashicorp. Additionally, many of them allow you to write your policies in OPA, which is open source, so you can still potentially use them even if you stop using the service. Sentinel doesn’t provide that option. Some of them might even offer support for -tvfars so you don’t have to edit your code to use multiple workspaces in a single folder.</p><p>I hope this was informative and provided a broad overview of infrastructure automation with Terraform Cloud. If you have any questions or comments, feel free to reach out via linkedin at <a href="https://www.linkedin.com/in/steven-staley/">https://www.linkedin.com/in/steven-staley/</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2853cbbb920" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/continuous-infrastructure-deployment-with-terraform-cloud-2853cbbb920">Continuous Infrastructure Deployment with Terraform Cloud</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring An OpenAPI/Swagger First Approach to Serverless Development on AWS]]></title>
            <link>https://itnext.io/exploring-an-openapi-swagger-first-approach-to-serverless-development-on-aws-d19f0e9ca257?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d19f0e9ca257</guid>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[open-api]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Tue, 23 Feb 2021 06:31:48 GMT</pubDate>
            <atom:updated>2022-06-19T07:51:26.508Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kKZT9Bw3a4Vr1bFTElA1uA.png" /></figure><p><em>Note: when I use the word “Serverless” with a capital “S” I’m referring to the framework, whereas when I use the word “serverless” with a lowercase “s” I’m referencing the overall methodology.</em></p><p>Since AWS unveiled Lambda in 2014, the close to zero downtime, scalability, and low maintenance overhead have made lambdas an attractive option for API development. If you’ve done any type of serverless development, there are two frameworks you’ve most likely heard of, and probably used for developing serverless APIs. Those would be the <a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html">AWS-developed SAM-CLI</a> and the <a href="https://www.serverless.com/">Serverless framework</a>. What these frameworks share is a declarative approach to building APIs with AWS’s API Gateway (along with other AWS resources but today we’re focused on APIs).</p><p>If you’re using SAM, part of your config may look like this (by the way, all the code for this demo can be found <a href="https://github.com/5t33/serverless-swagger-demo">here</a>):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf68524b787c2322dffe98b5812e8cad/href">https://medium.com/media/cf68524b787c2322dffe98b5812e8cad/href</a></iframe><p>whereas if you’re using Serverless, it might look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32a140fd019ab5ddd3a0344b797c02dc/href">https://medium.com/media/32a140fd019ab5ddd3a0344b797c02dc/href</a></iframe><p>These configs bring together and simplify an API Gateway/Lambda setup using less boilerplate. In the end, we have something that looks closer to what a developer might be used to when building an API: a list of endpoints along with their method types and query parameters. However, there is an alternative solution to declaring an API using API Gateway that many developers are already familiar with: OpenAPI.</p><h3>OpenAPI</h3><p>If you’ve ever worked with OpenAPI (or the API specification formerly known as Swagger) you might have noticed how similar these serverless config files look to an OpenAPI specification. <a href="https://swagger.io/">Swagger.io</a>, the creators of OpenAPI, provide a variety of helpful tools for generating APIs based on OpenAPI specs, much like how Serverless/SAM generate APIs with API Gateway (Wow that might have been the record for the amount of time “API” has been used in 2 sentences). API Gateway actually supports declaration via the OpenAPI spec as well.</p><p>An open API specification looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/488bf4c6ad9c25071ae96fa76d1a1423/href">https://medium.com/media/488bf4c6ad9c25071ae96fa76d1a1423/href</a></iframe><p>Looks quite similar to the two earlier examples, right? In this case, multiple tools have evolved into a similar format for a similar purpose, a phenomenon that occurs both in software and in biology (<a href="https://www.popsci.com/story/animals/why-everything-becomes-crab-meme-carcinization/">which is why everything evolves into a crab</a>). However, while serverless configs and OpenAPI may look similar, OpenAPI is actually a far superior way to declare APIs. With OpenAPI you get the benefit of a widely used spec as well as built-in, interactive documentation, which comes with a <a href="https://www.postman.com/">Postman</a>-like runner called <a href="https://swagger.io/tools/swagger-ui/">swagger-ui</a> in addition to an editor for your config called <a href="https://swagger.io/tools/swagger-editor/">swagger-editor</a> (by the way, you can also import your OpenAPI spec into postman if you like). The documentation looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4xYn6VJhr9TCdimKHM-Nw.png" /></figure><p>and the editor looks a bit like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2RLwVMfA5DyLF3lMWWLtw.png" /></figure><p>Additionally, many implementations of OpenAPI code-generation support validation of your spec out of the box. This means documentation and validation in the same place along with the nifty tools outlined above. I see this as an absolute win.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*4dpWG3j2mVttHAyvuGobfg.jpeg" /></figure><p>So, with this in mind, I’m going to spend the next few sections exploring an OpenAPI approach to serverless API development.</p><h3>API Setup</h3><h4>Serverless</h4><p>After a bit of experimentation, it became clear that both SAM and Serverless have limitations in regard to OpenAPI. Serverless doesn’t actually fully support declaring your API Gateway with OpenAPI. Instead, you declare parts of your OpenAPI spec inside the Serverless config along with your endpoints (using the <a href="https://github.com/deliveryhero/serverless-aws-documentation">serverless-aws-documentation</a> plugin), like so :</p><pre>custom:<br>  documentation: ${file(serverless.doc.yml):documentation}<br><br>functions:<br>  myFunc:<br>    events:<br>      - http:<br>          path: getStuff<br>          method: get<br>          documentation: ${file(serverless.doc.yml):endpoints.myFunc}</pre><p>While this isn’t the worst, it requires you to re-declare your endpoints, which is exactly what I’m trying to avoid. Additionally, it does not actually validate your requests locally with serverless-offline as API Gateway would. There is a package called <a href="https://github.com/RafPe/serverless-reqvalidator-plugin#readme">serverless-reqvalidator-plugin</a> but it requires you to declare your validators in the serverless file instead of in OpenAPI, which again defeats the purpose and doesn&#39;t allow for the use of swagger tooling.</p><h4>SAM</h4><p>SAM is a bit better about allowing you to simply declare your entire API with the OpenAPI config, like so:</p><pre>Resources:<br>  Api:<br>    Type: AWS::Serverless::Api<br>    Properties:<br>      Name:<br>        Fn::Sub: ${AppName}-api-${Stage}<br>        StageName: !Ref Stage<br>        DefinitionBody:<br>          Fn::Transform:<br>            Name: AWS::Include<br>            Parameters:<br>              Location: swagger.yml</pre><p>This will also work with SAM’s local command start api . You do still have to declare the functions in your SAM config, but there isn’t much of a way around that so I don’t consider it too much duplicate work. However, SAM doesn’t validate your input locally, as evidenced by this<em> </em><a href="https://github.com/aws/aws-sam-cli/issues/364"><em>issue from 2018</em></a>. I’m excited for that to be included as a feature once they get around to it.</p><p>Anyway, for the rest of this post, I’ll be using SAM to build and deploy my API. I will hopefully be able to work around the lack of local validation by writing tests against an OpenAPI validator. Considering that I would have written tests against my validation anyway, this isn’t a huge deal. Although to ensure that my validator is in line with API Gateway (it should be, but who knows), acceptance testing will be crucial, which is something I would have also done, but translating these tests from integration (against the validator package) to acceptance (against the deployed API) will require some logic. Anyway, let’s get started.</p><h4>OpenAPI config</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb871b46d05dc4c9ddda4d4b0ae10776/href">https://medium.com/media/fb871b46d05dc4c9ddda4d4b0ae10776/href</a></iframe><p>This creates one endpoint called /testFuncthat takes a single required string parameter. It also sets up request validation via the x-amazon-apigateway-request-validator. This validation is something you would normally have to set up in another CloudFormation resource. <strong>Something that’s rather important to note here</strong> is the discrepancy between the httpMethod parameter which reads POST and the OpenAPI Config method key which reads get: . The reason for this is that when using the lambda function integration with API Gateway you must list POST as the httpMethod, even if the endpoint type is a GET. This will not interfere with how you call your API, it only changes how API Gateway interacts with your lambda. I would think the docs would shout this out more loudly, but you can find it in the description of the “httpMethod” property <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-integration.html">here</a> in the OpenAPI extension docs as well as this <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/integration-request-basic-setup.html">example</a>.</p><p>By the way, on the subject of OpenAPI extensions, x-amazon-apigateway-request-validator is actually one of many <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions.html">OpenAPI extensions </a>AWS provides for declaring API Gateway resources or configurations, such as x-amazon-apigateway-any-method-object , x-amazon-apigateway-auth , and x-amazon-apigateway-cors . Together these should give you all you need to declare an API gateway with OpenAPI.</p><p>Now, the SAM config which will be using the OpenAPI config:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/372e75ddd4b43485e39e1d4996c24274/href">https://medium.com/media/372e75ddd4b43485e39e1d4996c24274/href</a></iframe><p>This SAM config will set up the API and the corresponding function with X Ray tracing enabled (to read more about X Ray, check out <a href="https://itnext.io/a-deep-dive-into-serverless-tracing-with-aws-x-ray-lambda-5ff1821c3c70?source=your_stories_page-------------------------------------">this deep dive</a>). Additionally, the config sets the “MethodSettings” and AccessLogSetting config values which turn on logging for API Gateway and format that logging into JSON respectively.</p><p>Now, run sam local start-api:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0SJrMb8iCmPh6FUrefqmmg.png" /></figure><p>and I can curl my endpoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dy5dEYOpet_nR3z3WqUZkQ.png" /></figure><p>Great, it works. This proves that we can use an OpenAPI config to run an API Gateway configuration locally, However, as you can see, the request is not being validated (the config specifies that a query param called testQuery be present). I mentioned earlier is a current issue with SAM, so that will require some testing to fill in the gap. Now on to deployment.</p><h3>Deployment</h3><p>To kick off a deployment for this demo API, run:</p><pre>sam deploy -t sam.yml --guided</pre><p>After initial deployment, you can run the command without the --guided flag, assuming that you picked the option to save your deployment config to a local file called samconfig.toml by default.</p><p>Now that the API is deployed, you should be able to navigate to the “dashboard” section of the API Gateway console in AWS to find your endpoint, and then test it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*aMabCa5bUz2OU-FgDwjLQA.png" /></figure><p>Which worked out great. We can see that the OpenAPI JSON-schema validation, while not available locally with SAM, is working as expected once deployed. Now let’s see if it correctly connects to Lambda once the variable is provided:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*XSIt6j_k6Tr8gKQ8Q5waAg.png" /></figure><p>Hah. Well, that’s not right. Let’s take a look at what might be the issue. After testing with the API Gateway test option in the console, I see the following logs (which are available because they were turned on in the SAM config):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*CLR_pKAtW1sigjD0sHum1A.png" /></figure><p>It looks like the API doesn’t have permission to talk to the lambda. After a bit of experimentation, I noticed this rather important difference between declaring an API with SAM vs an OpenAPI config. When deployed with SAM, your lambdas will be provided with a resource-policy, that will look something like this:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Id&quot;: &quot;default&quot;,<br>  &quot;Statement&quot;: [<br>      {<br>        &quot;Sid&quot;: &quot;demo-swagger-sam-TestFunctionTestFucPermissionProd-1LLU5CLKO2NJ0&quot;,<br>        &quot;Effect&quot;: &quot;Allow&quot;,<br>        &quot;Principal&quot;: {<br>          &quot;Service&quot;: &quot;apigateway.amazonaws.com&quot;<br>        },<br>        &quot;Action&quot;: &quot;lambda:InvokeFunction&quot;,<br>        &quot;Resource&quot;: &quot;arn:aws:lambda:us-east-2:01234567890:function:demo-swagger-sam-test-func-dev&quot;,<br>        &quot;Condition&quot;: {<br>        &quot;ArnLike&quot;: {<br>          &quot;AWS:SourceArn&quot;: &quot;arn:aws:execute-api:us-east-2:01234567890:4gic1pfddb/*/GET/testFunc&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>However, when deploying with an OpenAPI config the lambda resource-policy (found under the “permissions” tab of the Lambda console) is empty. This is (again) ratified by the OpenAPI <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-integration.html">x-amazon-apigateway-integration</a> extension docs, under the “credentials” section:</p><p>“For AWS IAM role-based credentials, specify the ARN of an appropriate IAM role. <strong>If unspecified, credentials default to resource-based permissions that must be added manually to allow the API to access the resource</strong>. For more information, see <a href="https://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html#intro-permission-model-access-policy">Granting Permissions Using a Resource Policy</a>.”</p><p>So, our options are to either add a resource policy to each lambda allowing the API Gateway resource to invoke them, or add an execution role to the API Gateway with permission to invoke each of our lambdas. I prefer the second option since it will be localized to one resource. So, to accomplish this, I will be adding the following IAM role &amp; policy resource to the SAM config:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f632d6c38d03b92a1b59abb8f9aacc89/href">https://medium.com/media/f632d6c38d03b92a1b59abb8f9aacc89/href</a></iframe><p>Which will be used in the OpenAPI spec via the credentialsx-amazon-apigateway-integration option like so:</p><pre>x-amazon-apigateway-integration:<br>  credentials:<br>    Fn::Sub: ${ApiRole.Arn}</pre><p>Now, after re-deploying the API and calling the test endpoint using the expected query parameter, I see the expected output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmZNP6hHy8eADy_PE4Swtg.png" /></figure><p>Now on to the swagger tooling.</p><h3>Tooling</h3><h4>Swagger-Editor</h4><p>To run the editor there are directions for both the docker image and the npm package <a href="https://swagger.io/docs/open-source-tools/swagger-editor/">here</a>. I’ll be using the docker image. To get that up and running, run the command:</p><pre>docker run -p 8080:8080 -v $(pwd):/tmp -e SWAGGER_FILE=/tmp/swagger.yml swaggerapi/swagger-editor</pre><p>This <em>should</em> start up the swagger editor using my local OpenAPI spec (swagger.yml) on port 8080.</p><p>Aaand success!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_phqP446W2LR4RXWIxfgqA.png" /></figure><p>As you can see from the above image, the swagger editor is able to interpret the OpenAPI config. And, it even understands the API Gateway extensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5gqg5W1cat_7BDq2tdq-g.png" /></figure><p>So that’s pretty cool. To illustrate how this works, I’ll update the swagger to include a new endpoint, via the UI. First, by clicking “insert”, which gives me the following dropdown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*HgvsHRDH-hgcpYcicIISOA.png" /></figure><p>From which I select “Add Path Item” Which pops up a dialogue box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAKTmj8tRhd0AuCKRsZ12g.png" /></figure><p>which I can fill out to insert a new path. After this, I would then be able to add an operation, info, etc. One thing I found odd is that it doesn’t give me the option to add any schemas. It might be that the “components” and “schemas” section has to already exist to edit, which would be odd, so I’ll try adding some random schemas like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a7f12d285684edc47a7340260170ed6/href">https://medium.com/media/4a7f12d285684edc47a7340260170ed6/href</a></iframe><p>This results in the schemas being listed on the right:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFeeWQ9HeBlh9-JUZFynOw.png" /></figure><p>But still no option to add components and/or schemas to the OpenAPI spec via the “insert” button. This is a bit annoying, but I was probably just going to edit the swagger spec by hand anyway, so the editor was more of a “nice to have”. That being said, it looks like <a href="https://app.swaggerhub.com/">swaggerhub</a> gives you a different version of the editor that does allow for adding schemas and the like (you can see the little “+” toggle next to “Schemas” on the left):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xq-qCcYQxObDsEvGK_Fa-g.png" /></figure><p>However, for some reason, you aren’t able to add endpoints using the swaggerhub editor. Additionally, the image doesn’t actually allow you to save to your swagger spec’s location. So it seems like editing the document with the GUI is out of the picture so from here on out I’ll just be editing it by hand.</p><p>For now, on to the more important bit, the Swagger UI:</p><h4>Swagger UI</h4><p>The swagger UI is a bit more important than the editor because it will be used as live documentation. Therefore, as opposed to the editor which was launched as a utility to be used locally, the UI will need to be deployed along with the API. This will help locally as well because it will be running on the same port, which will avoid cors errors when calling the endpoints (which would be solvable, but why go to the trouble?).</p><p>To deploy the UI I’ll be using <a href="https://www.npmjs.com/package/express">express</a> along with the <a href="https://www.npmjs.com/package/swagger-ui-express">swagger-ui-express</a> package and <a href="https://www.npmjs.com/package/@vendia/serverless-express">@vendia/serverless-express</a> (which is the new location for the now depreciated <a href="https://www.npmjs.com/package/aws-serverless-express">aws-serverless-express package</a>). That lambda will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc16faf0f0719d3295af7f771887514c/href">https://medium.com/media/bc16faf0f0719d3295af7f771887514c/href</a></iframe><p>To add this new function, I’ll be adding two additional paths to my OpenAPi config:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea8a35587fd11212849eeb52dfeab5e9/href">https://medium.com/media/ea8a35587fd11212849eeb52dfeab5e9/href</a></iframe><p>in addition to updating the swagger config to include this new function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64db9735af7ea93d12bac457ab20ad0a/href">https://medium.com/media/64db9735af7ea93d12bac457ab20ad0a/href</a></iframe><p>and adding to the ApiRole policy’s “Resource” section to include both lambda ARNs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/105808817d59d75f3ba93e16520c58a7/href">https://medium.com/media/105808817d59d75f3ba93e16520c58a7/href</a></iframe><p>Now, I can run sam local start-api -t sam.yml and visit localhost:3000/docs to see the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q0z2Ax0ztmtn8Ns4DncILQ.png" /></figure><p>To break this down a bit, the /docs route will direct requests to the docs.js lambda function previously discussed. However, a second option is listed under the /docs option called /{proxy+} . If you haven’t seen the {proxy+} option before, this is an API Gateway-specific configuration for a catch-all route (to be used with a proxy service, e.g. express). The reason for this is the swagger-ui-express middleware actually serves a set of static resources which are re-directed by the call to “/docs”.</p><p>As you can see in this network output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jc24-4pFgWRVTyDtdPRk5A.png" /></figure><p>However, since this proxy option will send all requests to your /docs endpoint if they don’t expressly (no pun intended) match any of your API Gateway’s other endpoints, it will be a good idea to add a 404 catch-all to docs.js. I was under the impression that one would be able to set a proxy like this: /docs/{proxy+} and therefore only proxy requests along a certain path (so that API Gateway could handle the 404/403s) but it doesn’t look like that’s actually possible from testing.</p><p>Anyway, after deployment we should be able to get the same thing from our API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5NtjmM-DWVdIK96UoeoHA.png" /></figure><p>Oops. Guess Not.</p><p>This is due to the interaction of API Gateway and Express. API Gateway prepends “/dev” to the API, and the Express API redirects from “/dev/docs” to “/docs/” to retrieve the static assets. Since “/dev” isn’t part of the base url, the browser removes the “/dev” as part of the redirect, as you can see here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxBCzm0k2imwXr4WXeIV2w.png" /></figure><p>While this is annoying, it’s actually not a huge issue because if this were running this as a real API, it would probably have a custom domain name anyway, so this wouldn’t be a poblem. The “/docs” redirect would navigate to the base of the custom domain, which wouldn’t have “dev” prepended to it. So to fix this I’m going to create a quick domain in Route53 and a cert in ACM and then add them to the SAM API resource config like so:</p><pre>Domain:<br>  CertificateArn: arn:aws:acm:us-east-2:0123456789012:certificate/&lt;cert-id&gt;<br>  DomainName: &quot;serverless-swagger-demo.com&quot;<br>  Route53:<br>    DistributionDomainName: serverless-swagger-demo.co<br>    HostedZoneId: &lt;zone-id&gt;</pre><p>And now I can view the docs endpoint via the new domain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FSxeqXDIhjXMv-7bleQvIA.png" /></figure><p>If you’re following along, you’ll notice that the naked domain <a href="https://serverless-swagger-demo/">https://</a>your-domain doesn’t work. Unfortunately, you’ll need to actually include the www. . I’m sure I’ve either misconfigured the SAM config (or the SAM config just doesn’t handle it, which wouldn’t surprise me) or I just need to add a CNAME to redirect to the root www. record. However, that’s not really the point of thist post, so I’m not going to worry about it.</p><p>Once the docs are up you can try out your endpoints and view schemas all from the UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Fd_BsMgT0RO3anqBWtduQ.png" /></figure><p>As a final bit of housekeeping it’s worth noting that you may not want these docs to be public, and so you may want to use API Gateway’s <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-apis.html">private API</a> option. However, API Gateway doesn’t allow for only part of your API to be public, so your documents would have to be served by a separate API. This would require some cors adjustments to actually call your endpoint from the docs. Alternatively, you could place a cognito authorizer in front of your docs as was done in <a href="https://betterdev.blog/serverless-swagger-ui-for-api-gateway/">this post</a> (the author decided to deploy to s3 instead of using API Gateway for the SwaggerUI, but it doesn’t make much of a difference).</p><h3>Testing</h3><p>As I mentioned earlier, without being able to test out validation manually with sam local start-api it will be extra important to write tests that check validation is working correctly. To do this I’ll be writing tests against an express OpenAPI validator plugin (specifically <a href="https://www.npmjs.com/package/express-openapi-validator">this one</a>) and using <a href="https://www.npmjs.com/package/supertest">Supertest</a> as if I were integration testing a regular express API. Theoretically validating an OpenAPI schema is just JSON validation, so API Gateway and any validator package should return the same results. That being said, running these same tests as acceptance tests after deploying the API will help to ensure this is the case.</p><p>This snippet of spaghetti code I quickly put together using <a href="https://www.npmjs.com/package/jest">Jest</a> should give a general idea of how that might work:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/859a4fc62e3d95317f38e08056e925de/href">https://medium.com/media/859a4fc62e3d95317f38e08056e925de/href</a></iframe><p>As you can see, there’s a bit of transformation going on in the “request” function to normalize output whether or not the request is to local Supertest or the remote API with Axios. In this way they can be run locally as integration tests using the command:</p><pre>stage=dev_local jest ./test/integration/* --testTimeout 10000</pre><p>or the same tests can be run as acceptance tests against the remote api with the command:</p><pre>base_url=https://www.serverless-swagger-demo.com stage=dev jest ./test/integration/* --testTimeout 10000</pre><p>Like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kgTsSqIcUGFCPeFbBoco0w.png" /></figure><p>I added a catchall “200” response, however you might need to add aditional options to simulate the response you’ll recieve from the actual API, e.g. 201, 301, etc.</p><h3>Conclusion</h3><p>I hope this post made you consider declaring your serverless APIs with OpenAPI instead of using the Severless/SAM config options. With the addition of the Swagger UI and built in validation by API Gateway, this seems like an attractive approach. There are a few gotchyas, but none seemed inconvenient enough to prevent me from going this direction in the future. Once the SAM CLI implements validation it will be even easier to write tests and test locally against an OpenAPI declaration. If you have any questions about the content of this post, serverless development, or software development in general feel free to reach out to me via <a href="https://www.linkedin.com/in/steven-staley/">linkedin</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d19f0e9ca257" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/exploring-an-openapi-swagger-first-approach-to-serverless-development-on-aws-d19f0e9ca257">Exploring An OpenAPI/Swagger First Approach to Serverless Development on AWS</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deep Dive into Serverless Tracing with AWS X Ray & Lambda]]></title>
            <link>https://itnext.io/a-deep-dive-into-serverless-tracing-with-aws-x-ray-lambda-5ff1821c3c70?source=rss-50d706bb696d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ff1821c3c70</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-x-ray]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Steven Staley]]></dc:creator>
            <pubDate>Mon, 31 Aug 2020 00:23:29 GMT</pubDate>
            <atom:updated>2021-05-30T23:39:41.363Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rjtRUAzmuhUO89BLAlVkvw.jpeg" /></figure><p>Over the past few weeks I’ve been experimenting with building a Serverless API on AWS with the goal of having everything needed to run a production system. One necessary piece was distributed tracing. While I’d seen a bit of what some non-AWS options had to offer, the extra cost of the services themselves, along with actually getting the data to them, was a bit prohibitive for what I imagined should be possible (and cheaper) with only AWS, which brought me to X Ray. So, I wanted to see if I could use AWS X Ray to get everything I wanted out of a tracing solution. First, I’ll set some expectations of what I need in a production tracing solution as a goal to orient the research process.</p><h3>Edit 02/20/21</h3><p>A few months after releasing this post I was alerted to something I missed while experimenting with X Ray’s tracing sampling rules (shout out to <a href="https://twitter.com/vishnuprasad73">Vishnu Prasad</a> for reaching out to me). While you seem to be able to increase sampling rate for API Gateway, Lambda is actually fixed at 1 request per second and 5% anything above that as per <a href="https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html">this documentation</a>. Apparently, the ability to increase sampling rates for Lambda is <a href="https://forums.aws.amazon.com/message.jspa?messageID=941643">in the works with no current release date</a>. Unfortunately, this makes X Ray fairly useless for serverless development outside of hobby projects. Cloudwatch logs are great but as a tracing solution, 1 request per second is extremely low throughput. Luckily there are some alternatives such as <a href="https://lumigo.io/">Lumigo</a> and <a href="https://epsagon.com/">Epsagon</a> that seem to be taking on this challenge.</p><h3>Requirements</h3><ol><li>Distributed tracing. On a high level, I want to see the path of a request from service to service, along with location of errors and details on latency.</li><li>Internal tracing, or a “flame graph”. I want to be able to see the breakdown of a request inside of a lambda, along with a stack trace when an error occurs. This will also help me spot any parts of my lambda that are running particularly inefficiently.</li></ol><h3>Setup</h3><p>First, a standard Serverless project setup and code walk-through. By the way, all the code I used in this post is available <a href="https://github.com/5t33/production-mock-serverless-api-demo">here</a>.</p><pre>npm i -g serverless<br>sls create --template aws-nodejs --path lambda-demo-api<br>cd lambda-demo-api &amp;&amp; npm init &amp;&amp; npm i --save-dev serverless-offline</pre><p>The file structure of the lambda we’re going to deploy looks like this:</p><pre>├── anotherFile.js<br>├── index.js<br>├── logger.js<br>├── node_modules<br>├── package-lock.json<br>├── package.json<br>├── serverless.yml<br>└── utils.js</pre><p>I’ll be explaining each of these files later on. For now, let’s check out the initial serverless config.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d62e08cfffb67d20ab8af21f37886c1c/href">https://medium.com/media/d62e08cfffb67d20ab8af21f37886c1c/href</a></iframe><p>To test out different configuration options I’ve added three endpoints: a health check that returns a 200 status code, a second that returns a 400 status code, and a third that throws an error and returns a 500 status code; three typical API responses. The idea here is that we should be able to isolate api calls that return 500s, along with the stack traces that accompany them, via our tracing solution (X Ray). I’ve added lambda tracing under the provider which turns on X Ray for each lambda, a more recent option that used to require a plugin.</p><p>When using the default global role for all of your functions, Serverless automatically provides permissions for Cloudwatch, however when providing function specific roles you need to give them access to Cloudwatch yourself. I also provided a few permissions that allow the lambda to connect to X Ray for tracing.</p><p>Additionally, I’ve downloaded the <a href="https://github.com/dherault/serverless-offline">serverless-offline</a> plugin, which is helpful for local development using the Serverless framework. It’s saved as development dependency so it won’t be included in the deployed node modules. Any additional dependencies will increase cold start time, so it’s important that you’re diligent about keeping your dependencies separated.</p><p>On line 10 and 34 we use the ${opt: stage} option to accesses the value provided when you run:</p><pre>sls deploy --stage &lt;test | dev | staging | prod&gt;</pre><p>With this setup Serverless will throw an error if you don’t include this flag. Forcing you to explicitly declare your stage helps to prevent incorrect deployments. Serverless includes your stage in the name of each of your deployed resources, a small but useful feature that helps when searching for resources, which isn’t available with AWS SAM or at least not by default.</p><p>You can find all my code for this initial set up in this <a href="https://github.com/5t33/production-mock-serverless-api-demo/tree/23eb0b174f7a75967555634ade51cbcc643dabf2">commit</a>. Commits to this repo will follow the code updates in this article, but I’ll also be going over each file. The initial handler for the mock API looks like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/780ba3d88642f80dfa2690cfe4dc9e51/href">https://medium.com/media/780ba3d88642f80dfa2690cfe4dc9e51/href</a></iframe><p>As you can see, we’re doing some routing that you may find redundant considering we’re using Severless. We <em>could</em> use our Serverless config to route each endpoint to a different function, so we wouldn’t have to do these conditionals. I could write a whole post on this, so for now I’ll just say that it’s mostly personal preference. Still, the if-block is a bit ugly and there’s a better options for internal routing that I’ll get into later on, but it’ll work for now. Let’s get into the other files I’m importing in the handler.</p><p>First “utils.js”:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a07db6b8c8982e8ac52ef36ce8cc253b/href">https://medium.com/media/a07db6b8c8982e8ac52ef36ce8cc253b/href</a></iframe><p>Above are a few simple functions that centralize the response format of the mock API. First, a class that extends the native JS Error class called “ApiError”. The purpose of this is to differentiate between custom/expected errors and unexpected exceptions, along with giving the option to provide a status code and custom response body. The handleError function on lines 21–30 is where the current error’s class is checked against the custom error class. Finally, there’s formatError and formatResp, which standardizes output and header format.</p><p>Finally, “anotherFile.js”</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cff0c32f27bc23cf050b5744054886e/href">https://medium.com/media/3cff0c32f27bc23cf050b5744054886e/href</a></iframe><p>These functions simulate the 3 API responses I mentioned earlier. I’m putting them in another file to make the stack trace a bit meatier when we get there.</p><p>Now it’s time to run it locally:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8T_-bqT1MKxGZ78pBZkmsw.gif" /></figure><p>And here are the responses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*brrEa60h7WP6I_mFkIdbnw.gif" /></figure><p>Sweet, everything works. Now let’s deploy and mess around with it on AWS.</p><pre>sls deploy --stage dev</pre><h3>X Ray</h3><p>I’ll spare you the deployment output, but after deployment we get the 3 urls and can now send some requests. When hitting each endpoint as we did locally above we get the expected output and status codes. We can check out X Ray to see what the tracing looks like by navigating to the lambda section of the AWS console and clicking “View Active Traces” under AWS X Ray. the X Ray dashboard looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTFw81dadqNCgyx85eWLWw.png" /></figure><p>The endpoints were hit 9 times, with an average response time of 79.7 ms. Cool! But something’s a bit odd. Why are all of the responses 200 OK? I hit the “/dev/throw_error” endpoint several times so that shouldn’t be the case. Additionally, we don’t have any insight into the specific endpoints we hit, which isn’t very helpful. Let’s see how the tracing turned out by selecting one of the blue, hyper-linked IDs under the trace list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ErfqAS99FufTtxk2KR27yQ.png" /></figure><p>Interesting. We can see the invocation and lambda function, but we don’t see anything pertaining to the specific functions we called inside the lambda, or even the handler. There are a few issues here, but one that’s somewhat obvious is that we didn’t set up <a href="https://www.serverless.com/blog/framework-release-v141">API Gateway X Ray integration in our Serverless config</a>. This is a relatively easy one-line fix. We just need to add “apiGateway: true” to the tracing section of the “provider” block of the serverless config, like so (The code for this is <a href="https://github.com/5t33/production-mock-serverless-api-demo/tree/88cf383291cf8cc792440b348d27aa705c5f0804">here</a>):</p><pre>provider:<br>name: aws<br>runtime: nodejs12.x<br>stage: ${opt:stage}<br>region: us-east-2<br>tracing:<br>  lambda: true<br>  apiGateway: true</pre><p>But for some CloudFormation related reason you cannot add API Gateway tracing without completely deleting the stack so we have to run the following:</p><pre>sls remove --stage dev &amp;&amp; sls deploy --stage dev</pre><p>After trying the new endpoints and navigating back to the X Ray dashboard, we can see an immediate improvement. Not only does it now have the expected status codes, but our responses are grouped by endpoint as well. Nifty. We can also see the client’s IP address, which I blocked for privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-gHYQ47rF6K4VEtneBHU8A.png" /></figure><p>Clicking on the trace of the 500 response, you can see that we have a bit more info here as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M9XfsCvhz72xTi7ON_SEXA.png" /></figure><p>The API Gateway is now in the trace map and the flame graph. However, we still don’t have any internal tracing of functions called inside of the lambda, but we’ll get to that in a second. You can also now see that an error occurred, although it isn’t an “error”, which specifies that a 400+ code was returned. It’s a “fault” indicating a 500+ status code. Upon clicking for details and navigating to the “exceptions” tab, we can see that our fault unfortunately doesn’t have a stack trace, which was one of the requirements I mentioned earlier (see below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dt21cuQi_s28lTOLTJc6xw.png" /></figure><p>To understand why this is, we need to look at how lambda and X Ray work together. In order for X Ray to report an error it actually requires your lambda to fail (at least as long as we aren’t using the SDK which I’ll get to in a second), which kind of makes sense when you think about it. The only information X Ray has to go on is what you return from your lambda. So if you return a resolved promise or normal response without an actual error object, there’s no reason for it to report an error and nothing to report if it did. The format of response from a failed lambda looks like this:</p><pre>{<br>  &quot;errorType&quot;: &quot;...&quot;,<br>  &quot;errorMessage&quot;: &quot;...&quot;,<br>  &quot;trace&quot;: [<br>    &quot;...&quot;<br>  ]<br>}</pre><p>X Ray needs that error output to provide the stack trace (without using the SDK which I’ll be getting into later on). In order to fail, the lambda will need to either use the context.fail function, the first argument of the provided lambda callback function, (e.g. callback(error)), or in the event of an async lambda, return a rejected promise. Let’s see what happens when we reject instead of resolving in the error handler function in the case of a 500. The utils.handleError function will now look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ce5cac789309c319e0ecceab9fb4018/href">https://medium.com/media/6ce5cac789309c319e0ecceab9fb4018/href</a></iframe><p>After re-deploying, we see the following Postman output from the throw_error endpoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pi4hfDpaqC-ozlzozPJyXA.png" /></figure><p>The response code is no longer 500, and the error message is now JSON. What happened? The error message that we’re now seeing isn’t actually the error message sent from the API response (as I mentioned earlier, we’re rejecting with the error itself now, not using the formatError function). A 502 response indicates an error thrown down stream of a proxy server which routed our request. This makes sense, as the API Gateway was that proxy server.</p><p>There’s now an error visible as well as a fault in the trace. And when checking the exception details we can now see a stack trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XiO5BNQKn8BOhdD39wUbsQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Or-trdN1DkMt8BYJQKQrKA.png" /></figure><p>Awesome! This completes the first requirement. Internal tracing still isn’t set up, but now we can trace a request all the way to any error that occurred and immediately find a stack trace without digging around in logs.</p><p>One drawback is not being able to respond with different 5XX errors to indicate varying issues. With this setup a lambda has to fail to provide X Ray with a trace, so we’re stuck with the 502 API Gateway response upon an unexpected error. However, there are two ways to get around this. One involves using the “Lambda Integration” instead of the “Lambda Proxy integration” with API Gateway, which uses a combination of regex and mapping templates to generate your response. You can read more about that <a href="https://medium.com/@lakshmanLD/lambda-proxy-vs-lambda-integration-in-aws-api-gateway-3a9397af0e6d">here</a>. The other option is to provide an error via an X Ray segment using the SDK, which is much easier. I’ll be going over that in the next section.</p><h4>Sampling</h4><p>Before going further it’s extremely important to note that by default X Ray does not capture every request. The default algorithm is <a href="https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html">1 request per second, and 5% of requests after that</a>. Here’s an image of traces captured from an endpoint I hit 100 times without any delay:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJCcv3qBYQnejqOK06Da-g.png" /></figure><p>Sidenote, I’m not sure why some of my requests went through as http and some as https? I’m not sure whether it’s an error with Axios or API Gateway, but if anyone can figure it out let me know.</p><p>Regardless, only 46 traces captured. That’s less than half. It would be pretty annoying if you had an error and you couldn’t find the related trace, right? To fix this, you need to <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-console-sampling.html">change the sampling rules applied to your functions</a>. Here’s the default that you’ll find in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fABvHpTkp2FKEFC2JnFeVw.png" /></figure><p>And the one I just added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WdSSuD5DB4tNKksiGrxHFQ.png" /></figure><p>By changing the fixed rate to 100%, I would have thought that it would capture all of my requests. Unfortunately that only got me to 65 out of 100:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HaC2Hj3h8Ln8o0V4bevwrg.png" /></figure><p>It seems that if your initial requests/second limit is low enough, the fixed rate wont keep up. After some fooling around I was able to get 100% coverage with 100 requests/second. Obviously this will depend on your use case. Your API will probably be rate limited, so you’ll have some idea of how many requests/second you might be seeing. Alternatively, if you regularly get a massive amount of traffic, you may want to limit the amount of tracing that X Ray does to save money (which I think is the original intent of the sampling option).</p><p>The downside here is in the way that CloudFormations random support for important details seems to depend on which way the wind blows or what Jeff Bezos had for breakfast this morning, and does not seem to support sampling configs. The bright side is that your console changes don’t appear to be overridden by subsequent deploys, and that rule probably won’t change that much, so you can set it and forget it after your initial deploy. However, if you like Terraform, they do have an option for it <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/xray_sampling_rule">here</a>.</p><h3>Internal Tracing with X Ray</h3><p>Now that X Ray is set up we can see requests moving from API Gateway to lambda, and shortly to downstream AWS services as well. Let’s take a look at some of the X Ray functions which allow us to trace inside of the lambda, add metadata, and attach errors to our traces.</p><p>First, install the X Ray sdk which gives you access to these options. (<a href="https://github.com/aws/aws-xray-sdk-node/tree/master/packages/core">you can read more about the package here</a>)</p><pre>npm i aws-xray-sdk-core</pre><h4>Local Development</h4><p>A Quick note. The X Ray sdk doesn’t really play nice with serverless-offline. It expects a trace id when the lambda is run, or it throws this error:</p><pre>Error: Missing AWS Lambda trace data for X-Ray. Expected _X_AMZN_TRACE_ID to be set.</pre><p>Another error you may see running locally is the following:</p><pre>&quot;Error: Failed to get the current sub/segment from the context.</pre><p>One option that at least prevents these errors from causing your lambda to fail is this environment variable:</p><pre>AWS_XRAY_CONTEXT_MISSING=LOG_ERROR sls offline --stage dev</pre><p>This tells X Ray to log the error instead of throwing it. However, the log output is rather verbose and annoying. To turn it off you can feed X Ray your own logger and set the level low enough that it doesn’t show up during your tests, like this:</p><pre>f(process.env.NODE_ENV === &#39;local_dev&#39; ) {<br>  AWSXRay.setLogger(Logger(&#39;silent&#39;));<br>}</pre><p>If you go this route it’s important to import X Ray once in a utilities file and import from that file all other times, or some instances of X Ray won’t have the logger you provided.</p><h4>Segments</h4><p>First, it’s important to understand the concept of a ‘segment’. A segment is simply a portion of code measured by timespan. Segments are all part of a tree, with one root segment at the top and subsegments below it. A subsegment is simply a child of another segment. There are only slight functional differences between the two. There are two segment related functions to note: “addNewSegment” will generate a subsegment of the calling segment, and “addSubsegment” will add a segment as a child to any other segment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b56e624ee48fcad80b93742bc0edfeb/href">https://medium.com/media/2b56e624ee48fcad80b93742bc0edfeb/href</a></iframe><p>As you can see above, in this option you have to explicitly pass your segment down to the lower function and create a new child segment there. Here’s how it would look in the rejecting function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1e257cdfe6f15ea78c37aa419122686/href">https://medium.com/media/a1e257cdfe6f15ea78c37aa419122686/href</a></iframe><p>You <strong><em>must</em></strong> call “close” on all your segments, or none of your segments will show up and you’ll lose your stack trace (which is also why you want to also print logs instead of relying solely on X Ray). Another important thing to mention is that Lambda is special in that comes with an automagically generated root segment which you can access via AWXRay.getSegment. It’s immutable so you can’t add annotations or metadata to it, and none of the other Segment functions work when called, but you can use it as a parent for further subsegments (which function as normal segments).</p><p>Let’s see how these segments look in the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DU-N1euvBAPl6su-rs2-eA.png" /></figure><p>Awesome! We can see the three different outputs: 1 OK response (200), 1 error (400 +), and one fault (500 +), and now we can dive all the way into the functions that were called, see how long they took, and where an error occurred. Now the trace graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dLN-Qw0IC1vTU8nwX-PEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BCx8hkrnfHlfu3Qc2Iz5eg.png" /></figure><p>Now we have a deeper look at the overall breakdown of time spent in the function, and we can see where the error occurred.</p><h4>addError</h4><p>As you may have noticed in the above example, “addError” was called in the segment of the failing function. This solves the problem from before of needing to reject the lambda to send a stack trace in X Ray. Let’s see what the dashboard looks like if we resolve the promise now that the lambda doesn’t need to fail for a stack trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2rKGRd1WdmXF-9EFYWDlfQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14HbjGFUHzSVw19mcUHH7w.png" /></figure><p>The error is still there, and you’re able to get the stack trace, but X Ray no longer shows the lambda itself as failing, only the endpoint and the function.</p><h4>addAnnotation &amp; addMetadata</h4><p>There are several other functions that you may want to take advantage of that are part of the Segment object, two of which are “addAnnotation” and “addMetadata”. These allow you to add information to individual segments which you will be able to see in your X Ray dashboard. One key difference between the two is that you can search through X Ray annotations, but not metadata. Another is that Metadata allows you to store whole objects, while annotations are key value pairs. You can read more about searching through traces <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-console-filters.html#xray-console-filters-details">here</a>.</p><p>Here’s an example query for searching through all traces for failures in which the invocation that took less than one second and had annotation “a = b”:</p><pre>service(&quot;lambda-api-demo-dev-DemoLambda&quot;) AND duration &lt; 1 AND !ok AND annotation.a = &quot;b&quot;</pre><p>If you’d like to learn more about other X Ray concepts, you can read about them <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html">here</a>.</p><h4>captureFunc &amp; captureAsynFunc</h4><p>Two other functions you may want to take advantage of are “captureFunc” and “captureAsynFunc”. addNewSegment doesn’t work with promises, so “captureAsynFunc” fixes that. Below, you can see how I’m using captureAsyncFunc in “anotherFile.js” (by the way, all of this code is available in <a href="https://github.com/5t33/production-mock-serverless-api-demo/tree/915a4ebe98a2c44863701b8a2491cdf3ba1730be">this commit</a>).</p><p>Using addNewSubsegment with async/await is a bit cleaner, but I didn’t realize it was an option for async calls until after I wrote this part, so for the rest of this post I’ll be using captureAsynFunc.</p><p>Here is the updated handler with X Ray wrappers included:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8a9a334f1992192edaafa138bed64c5/href">https://medium.com/media/c8a9a334f1992192edaafa138bed64c5/href</a></iframe><p>And “anotherFile.js”:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a74c5b81e2fe34d879ba8fd28f873468/href">https://medium.com/media/a74c5b81e2fe34d879ba8fd28f873468/href</a></iframe><p>While you can’t see it above, I’m calling “close” in the handlers. You may be tempted to call addError in the errorHandler as well before logging the error, but that would report the error as having occurred in the error handler function instead of the one it actually happened in.</p><h4>A Few Words of Caution</h4><p>I’m going to pause before going further to add a few words of caution. While segments are pretty cool, I don’t necessarily recommend using them to this extent. Getting to this point was <em>extremely</em> invasive into the code. I also don’t love the idea of wrapping every single function I write, and the fact that you <em>must</em> call close on your async segments or your internal tracing will blow up is a lot of mental overhead. It’s almost like writing C.</p><p>However, it is pretty nice to be able to resolve your function with any status code you like and still report an error to X Ray. So, wrapping your function with a single segment for that purpose might be a reasonable limit of how you’d use this functionality if you didn’t want to wrap all of your function calls. With X Ray’s other features, tracing every single function call might not even be necessary, which you’ll see in the next few sections.</p><h4>captureAWS &amp; capturePostgres</h4><p>What I mean when I say “other features” are the functions X Ray provides to trace anything that your function reaches out to. Let’s look at three options: other AWS services, http requests, and databases. To set this up I’m going to create a few additional resources: a DynamoDB table, an SQS queue, an SNS topic, and another lambda function. This setup is contrived and overly complex for the task but it’s just to provide an example. I’m going to add an endpoint which will pull 5 values from the dynamo table and push them to an SNS. The SNS will push to SQS, where the other function will pull the data and send it to a slack channel. You can find the code for this section <a href="https://github.com/5t33/production-mock-serverless-api-demo/tree/d4e67509e3a42593e73c41f8bf107741baa647ce">here</a>. I generated the fake user data via <a href="https://randomuser.me/">https://randomuser.me/</a>, and I used <a href="https://stevensteam-workspace.slack.com/apps/new/A0F7XDUAZ-incoming-webhooks">this slack app</a> for the slack webhook. Once you deploy, you can seed the db with:</p><pre>npm run seed-dev</pre><p>I’ve also added two files to the “common” folder: dynamodb.js and sns.js. Each of them contain these two lines (you could probably also just call it once at the top of your handler):</p><pre>const AWSXRay = require(&#39;aws-xray-sdk-core&#39;)<br>const AWS = AWSXRay.captureAWS(require(&#39;aws-sdk&#39;));</pre><p>That’s all you need to do to provide tracing for any AWS resource you use. Once we call the new endpoint “/dev/send_data” and then check out the X Ray dashboard, we’ll see some new information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3X1shPtfDV-kpVzA1yNklA.png" /></figure><p>There are now segments for the DynamoDB scan as well as the 5 SNS calls. If we look at the trace map we can see that it’s changed a bit as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*nv1SIhai7hw6cWBQKO2Wzg.png" /></figure><p>The DynamoDB and SNS are now included, but wait a minute… Why don’t we see the SQS or the other lambda function? I’ll get to that in a second, first let’s look at the other function’s handler:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fcd6435c412bb5885bb5574b9225f32/href">https://medium.com/media/7fcd6435c412bb5885bb5574b9225f32/href</a></iframe><p>Those first 3 lines are the most important. In the same way that “captureAWS” wraps AWS resource calls, captureHTTPsGlobal wraps all outbound http(s) calls. If we look at the X Ray dashboard for this function we see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CrsnEFphqkMq6PLt6S8nQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58f8M1hZTaXYmB1K_4FBaw.png" /></figure><p>We can see the call to the slack endpoint in both the graph and the chart. However, here you still don’t see the SQS queue. Unfortunately, once you get past one degree of separation from your lambda, X Ray isn’t able to connect the trace, or at least from SNS to SQS to another lambda, as you can see in <a href="https://github.com/aws/aws-xray-sdk-node/issues/208">this issue on github.</a> AWS is apparently working on some updates that would include a fix for this, so we’ll just have to wait and deal with separate dashboards for now. Apparently you can at least get the sqs into your trace (like the poster of that issue did) but I wasn’t able to work out how.</p><p>At this point we’ve covered how to trace http requests and calls to other AWS services. However we still need to cover databases. You’ll need a VPC/RDS instance to follow along here, which I used Terraform for, so it isn’t in the repo, sorry. However, if you set them up on your own, you should note that I pull in security group ids, the vpc id, and the rds endpoint using SSM which you can also replicate. Once that’s ready you can use the following scripts to seed your database:</p><pre>npm run seed-rds-local<br>npm run seed-rds-dev</pre><p>Now, in the lambda the X Ray set up is fairly simple. Much like with the aws-sdk and http, you just wrap your database import in a function call, as you can see below (you can find added RDS code at <a href="https://github.com/5t33/production-mock-serverless-api-demo/tree/81ac39a111be284463ab318be3ee0c1069f4aeb4">this commit</a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6808c92bd449284480858abcde0fd23b/href">https://medium.com/media/6808c92bd449284480858abcde0fd23b/href</a></iframe><p>I’m using pg-promise in this example, so I’m calling capturePostgres before the import, much like how captureHTTPsGlobal was called before importing Axios. And, as you can see, we now have an SQL segment in the trace graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-6JF67V5mJ1-PApo3Q1hQ.png" /></figure><p>With these three approaches to capturing segments, manually wrapping functions doesn’t seem nearly as necessary unless you need a deep look into what might be causing performance issues.</p><h3>Express with Lambda &amp; X Ray</h3><p>At the beginning of this post I mentioned that there’s a better way to route requests internally on a per-lambda basis (if you don’t want to use API Gateway for all of your routes, that is). One option is Express. Using the <a href="https://www.npmjs.com/package/aws-serverless-express">aws-serverless-express</a> module, you can use express inside your lambda, which gives you routing, middleware, and automatic parsing of query parameters for the minor cost of setting up the routes and calling “listen” on cold starts. I know some might consider that an anti-pattern, but I don’t think it’s that big of a deal if you’re only loading a few routes for a single endpoint. However, using express means that we will have to update the handler code to accommodate express. See below (by the way you can find this code <a href="https://github.com/5t33/production-mock-serverless-api-demo/tree/0209bc3fdc52c5d75cfceef2aed68535f3566bb4">here</a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1ce853c47134e5be067e6a1a564556c/href">https://medium.com/media/c1ce853c47134e5be067e6a1a564556c/href</a></iframe><p>Using the <a href="https://www.npmjs.com/package/aws-serverless-express">aws-serverless-express</a> package, routing is now handled using express. In order to add a unique request id to the logger as we had before, I included it in a middleware function. There’s also an X Ray express middleware package called aws-xray-sdk-express, although it doesn’t seem to do much more than add a wrapping segment, which I’m using above instead of the manual “handler” segment. I thought that it would at least trace middleware functions, but that doesn’t seem to be the case. It could just be that it behaves differently inside of a lambda than it does inside of an express container.</p><p>One other small drawback of the x-ray middleware is that it isn’t made to be used in lambda so it creates two separate traces — one for your lambda invocation and one for your handler, as you can see below. This is probably due to the express plugin trying to make its own root segment. This seems to cause a small discrepancy between the measurements recorded. You can see what that output looks like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*a5P8PVYllY_BmB1iI174cg.png" /></figure><p>That isn’t really ideal and the middleware doesn’t really add much else, so I decided to go back to a single wrapper for the lambda, which looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc62d0b7487480efddda712db73a8ccc/href">https://medium.com/media/dc62d0b7487480efddda712db73a8ccc/href</a></iframe><p><em>A word of caution here</em>: Now that we’re returning the result of express, there’s a chance you could send a stack trace back to your client. This would only happen if an uncaught error occurred in middleware upstream of the error handlers. aws-serverless-express handles the error, but it resolves the promise with a 500, which sends the full error back.</p><h3>Conclusion</h3><p>That’s pretty much it. This post became a lot longer than I intended, but I think diving in and doing some experimentation was worth the effort. I didn’t discuss the monitoring available in the X Ray console, but I recommend giving it a quick look. One thing I’d like to see added to X Ray is an easy way to link traces with logging. It looks like there’s another service called “ServiceLense” which does this, but it can’t imagine it would be too difficult to add a button to the related log stream in an X Ray trace. Anyway, thanks for reading and I hope this helped you get started using X Ray in your Serverless APIs.</p><p>By the way, I’d like to shout out <a href="https://www.manning.com/livevideo/production-ready-serverless">this course</a> by Yan Cui, along with <a href="https://theburningmonk.com/">his blog</a>, two resources I come back to repeatedly for information on building serverless systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ff1821c3c70" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/a-deep-dive-into-serverless-tracing-with-aws-x-ray-lambda-5ff1821c3c70">A Deep Dive into Serverless Tracing with AWS X Ray &amp; Lambda</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>