<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nuatu Tseggai on Medium]]></title>
        <description><![CDATA[Stories by Nuatu Tseggai on Medium]]></description>
        <link>https://medium.com/@nuatu?source=rss-dfcee22e2d1b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QVWaSjXfuCGgpourohfXVg.png</url>
            <title>Stories by Nuatu Tseggai on Medium</title>
            <link>https://medium.com/@nuatu?source=rss-dfcee22e2d1b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:12:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nuatu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Lambda@Edge: Why Less is More]]></title>
            <link>https://medium.com/stacks-on-stacks/lambda-edge-why-less-is-more-3b6d13db1b20?source=rss-dfcee22e2d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b6d13db1b20</guid>
            <category><![CDATA[cdn]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[lambda-at-edge]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Nuatu Tseggai]]></dc:creator>
            <pubDate>Thu, 21 Feb 2019 22:21:20 GMT</pubDate>
            <atom:updated>2019-02-26T22:21:44.860Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*Q9aTXhG8ujaJQuPV.jpg" /></figure><p>Lambda@Edge is a compute service that allows you to write JavaScript code that executes in any of the 150+ AWS edge locations making up the <a href="https://aws.amazon.com/cloudfront/">Amazon CloudFront</a> content delivery network (CDN) service.</p><p>In this post, I’ll provide some background on CDN technologies. I will also build out an application stack that serves country-specific content depending on where the user request originates from. The stack utilizes a Lambda@Edge function which checks the country code of an HTTP request and modifies the URI to point to a different index.html object within an S3 bucket.</p><h3>TL;DR: Less time, Fewer resources, Less effort</h3><ul><li>CDNs are ubiquitous. Modern websites and applications make extensive use of CDN technologies to increase speed and reliability.</li><li>Lambda@Edge has some design limitations: Node.JS only, must be deployed through us-east-1, limitations on memory size differ between event types, etc.</li></ul><p>Read on for a working example alongside tips and outside resources to inform you of key design considerations as you evaluate Lambda@Edge.</p><h3>The best of both worlds: Lambda + CloudFront</h3><ul><li>Fully managed: no servers to manage and you never have to pay for idle</li><li>Reliable: built-in availability and fault-tolerance</li><li>Low latency: a global network of 160+ Points of Presence in 65 cities across 29 countries (as of early 2019)</li></ul><h3>A Use Case</h3><p>You have a website accessed by users from around the world. For users in the United States, you want CloudFront to serve a website with US market-specific information. The same is true for users in Australia, Brazil, Europe, or Singapore and each of their respective markets. For users in any country besides those mentioned above, you want CloudFront to serve a default website.</p><p>Stackery will be used to design, deploy, and operate this stack; but the Infrastructure as Code and Lambda@Edge concepts are valid with or without Stackery.</p><p>Check out <a href="https://aws.amazon.com/lambda/edge/">this link</a> to explore many of the other use cases such as:</p><ul><li>A/B testing</li><li>User authentication and authorization</li><li>User prioritization</li><li>User tracking and analytics</li><li>Website security and privacy</li><li>Dynamic web application at the edge</li><li>Search engine optimization (SEO)</li><li>Intelligently route cross origins and data centers</li><li>Bot mitigation at the edge</li><li>Improved user experience (via personalized content)</li><li>Real-time image transformation</li></ul><h3>Background: Need for Speed</h3><p>Traffic on the modern Internet has been growing at a breakneck rate over the last two decades. This growth in traffic is being fueled by nearly 4 billion humans with an Internet connection. It’s estimated that more than half of the world’s traffic is now coming from mobile phones and that video streaming accounts for <a href="https://www.ispreview.co.uk/index.php/2018/10/video-streaming-and-netflix-dominates-global-internet-traffic.html">57.69% of global online data</a> traffic. Netflix alone is responsible for 14.97% of the total downstream volume of traffic across the entire internet! The other half comes from web browsing, gaming, file sharing, connected devices (cars, watches, speakers, TVs) Industrial IoT, and back-end service-to-service communications.</p><p>To keep pace with this rate of growth, websites owners and Internet providers have turned to CDN technologies to cache web content on geographically dispersed servers at edge locations around the world. Generally speaking, these CDN’s serve HTTP requests by accepting the connection at an edge location in close proximity to the user (latency-wise), organizing the request into phases, and caching the response content so that the aggregate user experience is fast, secure, and reliable.</p><p>When done correctly, the result is a win-win. The end user can expect faster load times and the website owner ends up with a lighter load on the origin server. Furthermore, service providers also end up with a more efficient system since less packets are contending for precious bandwidth on the backhaul portion of the major telecommunications networks (ie: the intermediate links between the core network, backbone network, and subnetworks at the edge of the network).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*hrNaXNGnL2jQu2aM.jpg" /></figure><p>For more background, check out the <a href="https://www.cloudflare.com/learning/cdn/what-is-a-cdn/">What is a CDN</a> page from CloudFlare and this <a href="https://aws.amazon.com/cloudfront/features/">Amazon CloudFront Key Features</a> page from AWS.</p><h3>Lambda@Edge</h3><p>Lambda@Edge is a relatively new feature (circa 2017) of CloudFront which enables the triggering of Lambda functions by any of the following four CDN events.</p><p><strong>Viewer Request</strong></p><p>Edge Function is invoked when the CDN receives a request from an end user. This occurs before the CDN checks if the requested data is in its cache.</p><p><strong>Origin Request</strong></p><p>Edge Function is invoked only when the CDN forwards a request to your origin. If the requested data is in the CDN cache, the Edge Function assigned to this event does not execute.</p><p><strong>Origin Response</strong></p><p>Edge Function is invoked when the CDN receives a response from your origin. This occurs before the CDN caches the origin’s response data. An Edge Function assigned to this event is triggered even if the origin returns an error.</p><p><strong>Viewer Response</strong></p><p>Edge Function is invoked when the CDN returns the requested data to the end user. An Edge Function assigned to this event is triggered regardless of whether the data is already present in the CDN’s cache.</p><p>When deciding which CDN event should trigger your Edge Function, consider <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-how-to-choose-event.html">these questions</a> from the AWS Developer Guide, as well as additional clarifying questions from this <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/lambdaedge-design-best-practices/">helpful AWS blog post</a> in the “Choose the Right Trigger” section.</p><h3>Sample Source</h3><p>The source code for this project is available from <a href="https://github.com/nuatu/demo-lambda-edge-example/">my GitHub</a>.</p><h3>Template Generation</h3><p>I used the Stackery editor to lay out the components and generate a template:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*87lAbn7Q92sjOOP9.jpg" /></figure><p>The template is available in the Git repo as template.yaml.</p><p>This application stack is pretty straightforward: a CDN is configured to serve a default index.html from an S3 bucket and the CDN is also configured to trigger a Lambda@Edge function upon any Origin Request events. Origin Requests are only made when there is a cache miss, but in the context of this application stack, cache misses will be rare. The default TTL for files in CloudFront is 24 hours — depending on your needs, you can reduce the duration to serve dynamic content or increase the duration to get better performance. The latter will also lower the cost because your file is more likely to be served from an edge cache, thus reducing load on your origin.</p><p>Pay special attention to <a href="https://github.com/Nuatu/demo-lambda-edge-example/blob/master/template.yaml#L11-L12">lines 11–12</a> within the infrastructure as code template. These lines configure the CDN to cache based on the CloudFront-Viewer-Country header which is added by CloudFront after the viewer request event.</p><p>Also, note that <a href="https://github.com/Nuatu/demo-lambda-edge-example/blob/master/template.yaml#L23">line 23</a> which specifies the “Price Class 200” for the CDN (which enables content to be delivered from all AWS edge regions except South America). Price Class All is the most expensive which enables content to be delivered from all AWS regions (this is the default if no other price class is specified). Price Class 100 is the cheapest and only delivers content from United States &amp; Canada and Europe. For more information on pricing check out <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PriceClass.html">this link</a>.</p><h3>Lambda@Edge Function</h3><p>The Lambda@Edge function checks if the country code of the request is AU, BR, EU, SG, or US. If it is, the URI of the HTTP request is modified to point to a specific index.html object (such as au/index.html or us/index.html) within the S3 bucket. The default index.html object is served from S3 bucket if the country code is NOT one of the above five.</p><p>Here’s the complete function code: <a href="https://github.com/Nuatu/demo-lambda-edge-example/blob/master/src/edgeFunction/index.js">index.js</a></p><pre>&#39;use strict&#39;<br><br>exports.handler = async (event) =&gt; {<br>    const request = event.Records[0].cf.request<br>    const headers = request.headers<br><br>    console.log(JSON.stringify(request))<br>    console.log(JSON.stringify(request.uri))<br><br>    const auPath = &#39;/au&#39;<br>    const brPath = &#39;/br&#39;<br>    const euPath = &#39;/eu&#39;<br>    const sgPath = &#39;/sg&#39;<br>    const usPath = &#39;/us&#39;<br><br>    if (headers[&#39;cloudfront-viewer-country&#39;]) {<br>        const countryCode = headers[&#39;cloudfront-viewer-country&#39;][0].value<br>        if (countryCode === &#39;AU&#39;) {<br>          request.uri = auPath + request.uri<br>        } else if(countryCode === &#39;BR&#39;) {<br>          request.uri = brPath + request.uri<br>        } else if (countryCode === &#39;EU&#39;) {<br>          request.uri = euPath + request.uri<br>        } else if (countryCode === &#39;SG&#39;) {<br>          request.uri = sgPath + request.uri<br>        } else if (countryCode === &#39;US&#39;) {<br>          request.uri = usPath + request.uri<br>        }<br>    }<br>    console.log(`Request uri set to &quot;${request.uri}&quot;`)<br><br>    return request<br>}</pre><h3>Deployment</h3><p>Of course, Stackery makes it simple to deploy this application into AWS, but it should be pretty easy to give the template directly to CloudFormation. You may want to go through and whack the parameters like ‘StackTagName’ that are added by the Stackery runtime.</p><p>Once the deployment is complete, the provisioned CDN distribution will have a DNS address. I deployed this application to several different environments, one of which I have defined as staging. Here’s the DNS address of that distribution: <a href="https://d315q2a48nys0i.cloudfront.net/">https://d315q2a48nys0i.cloudfront.net/</a></p><p>Lastly, go to the newly created S3 bucket and add this default <a href="https://github.com/Nuatu/demo-lambda-edge-example/blob/master/html/index.html">index.html file</a> to the root of the S3 bucket. Then create the following 5 “folders” in the S3 bucket and be sure that they are lowercase: au, br, eu, sg, us. I put folders in quotes because it’s not technically a folder, but via the UI, S3 refers to them as folders and allows them to be created as such. Once each folder is created, add the respective index.html that I have saved in this /html directory within this github project (ie: For the au bucket, copy over the index.html that I have saved at /html/au/index.html). The AWS CLI is convenient for this type of copying/syncing, check out <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html">this link</a> for tips pertaining to managing S3 buckets and objects from the command line.</p><p>If I hit the DNS address of the CDN distribution from Portland Oregon, I see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*ImG1igYohS3bK9ej.jpg" /></figure><h3>See How it Appears to the Rest of the World</h3><p>GeoPeeker is a pretty nifty tool that allows you to see how a site appears to the rest of the world. Just go to <a href="https://geopeeker.com/fetch/?url=https%3A%2F%2Fd315q2a48nys0i.cloudfront.net%2F">this link</a> and geopeeker will show the site I’ve deployed as it appears to users in Singapore, Brazil, Virginia, California, Ireland, and Australia.</p><h3>Conclusion</h3><p>I encourage you to explore the shape of the request event object as well as the response event object both of which can be found at <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-event-structure.html#lambda-event-structure-request">this link</a>. At one point prior to finding this page, I was getting my wires crossed in terms of the values available on each object. Once I found it, I was able to instantly get back on track and hone in on the URI value that I wanted to modify.</p><p>An alternative implementation to the changing the URI is to change the host. In that scenario, I could have created a separate S3 bucket for the default site and separate S3 buckets for the index.html for each of the 5 countries, then upon each Origin Request, modify the host instead of the URI when I found a match. Perhaps I’ll do that in a follow on post to show the difference in the resulting Infrastructure as Code template and Lambda@Edge function.</p><p>The use case I covered is relatively approachable. More advanced use cases, such as securing sites and applications from bots or DDOS attacks, would be really interesting and fun to implement using Lambda@Edge. It would be great to see more blog posts and/or reference implementations based on reproducible Infrastructure as Code samples that show <a href="https://hackernoon.com/postcards-from-lambda-the-edge-11a43f215dc1">Lambda@Edge based solutions</a> that target A/B testing, analytics, and user authentication and authorization. Let me know <a href="https://twitter.com/dataface?lang=en">on twitter</a> or in the comments which types of use cases your interested in and I’ll work to put them together or coordinate with various serverless experts to bring the solutions to life.</p><p><em>Originally published at </em><a href="https://www.stackery.io/blog/lambda-at-edge/"><em>www.stackery.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b6d13db1b20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stacks-on-stacks/lambda-edge-why-less-is-more-3b6d13db1b20">Lambda@Edge: Why Less is More</a> was originally published in <a href="https://medium.com/stacks-on-stacks">Stacks on Stacks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Disaster Recovery in a Serverless World — Part 3]]></title>
            <link>https://medium.com/stacks-on-stacks/disaster-recovery-in-a-serverless-world-part-3-1a4a735fcf08?source=rss-dfcee22e2d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a4a735fcf08</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Nuatu Tseggai]]></dc:creator>
            <pubDate>Thu, 18 Oct 2018 22:56:57 GMT</pubDate>
            <atom:updated>2018-10-18T22:56:57.084Z</atom:updated>
            <content:encoded><![CDATA[<h3>Disaster Recovery in a Serverless World — Part 3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*KDVD3G1xLP6Xa7cf.jpg" /></figure><p><em>This is part three of a multi-part blog series. In the first post we covered </em><a href="https://www.stackery.io/blog/disaster-recovery-in-a-serverless-world-1-of-2/"><em>Disaster Recovery planning when building serverless applications</em></a><em> and in the second post we covered </em><a href="https://www.stackery.io/blog/disaster-recovery-serverless-2-of-2/"><em>the systems engineering needed for an automated solution in the AWS cloud</em></a><em>. In this post, we’ll discuss the end-to-end Disaster Recovery exercise we performed to validate the plan.</em></p><p>The time has come to exercise the Disaster Recovery plan. The objective of this phase is to see how closely the plan correlates to the current reality of your systems and to identify areas for improvement. In practice, this means assembling a team to conduct the exercise and documenting the process within the communication channel you outlined in the plan.</p><h4>Background</h4><p>In the first post you’ll find an outline of the plan that we’ll be referencing in this exercise. Section 2 describes the process of initiating the plan and assigning roles and Section 3 describes the communication channels used to keep stakeholders in sync.</p><h4>Set the Stage</h4><p>Decide who will be involved in the exercise, block out a chunk of time equivalent to the Recovery Time Objective, and create the communication channel in Slack (or whatever communication tool that your organization utilizes; ie: Google Hangouts, Skype, etc).</p><p>Create a document to capture key information relevant to the Disaster Recovery exercise such as who is assigned which roles, the AWS regions in play, and most importantly, a timeline pertaining to the initiation and completion of each recovery step. Whereas the Disaster Recovery plan is a living document to be updated over time, the Disaster Recovery exercise document mentioned here is specific to the date of the exercise; ie: Disaster Recovery Exercise 20180706.</p><h4>Conduct the Exercise</h4><p>Enter the communication channel and post a message that signals the start of the exercise immediately followed by a message to the executive team to get their approval to initiate the Disaster Recovery plan.</p><p>Upon completion of the exercise, post a message that signals the end of the exercise.</p><h4>Key Takeaways</h4><ol><li>Disaster Recovery exercises can be stressful. Be courteous and supportive to one another.</li><li>High performance teams develop trust through practicing to communicate effectively. Lean towards communication that is both precise and unambiguous. This is easier said than done but gets easier through experience. Over time, the Disaster Recovery plan and exercise will become self reinforcing.</li><li>Don’t forget to involve a representative of the executive team. This is to ensure that the operational status is being communicated across all levels.</li><li>Be clear on which AWS region represents the source and which AWS region represents the DR target.</li><li>Research and experiment with the options you have available to automate DNS changes. Again, the key here is gaining the skills and confidence through testing and optimizing for fast recovery. Know your records (A and CNAME) and the importance of <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-values-basic.html#rrsets-values-basic-ttl">TTL</a>.</li><li>Verify completion of recovery from the perspective of a customer.</li><li>Conduct a retrospective to illuminate areas for improvement or research. Ask questions that encourage participants to openly discuss both the technical and psychological aspects of the exercise.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a4a735fcf08" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stacks-on-stacks/disaster-recovery-in-a-serverless-world-part-3-1a4a735fcf08">Disaster Recovery in a Serverless World — Part 3</a> was originally published in <a href="https://medium.com/stacks-on-stacks">Stacks on Stacks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Disaster Recovery in a Serverless World — Part 1]]></title>
            <link>https://medium.com/stacks-on-stacks/disaster-recovery-in-a-serverless-world-part-1-72d666f80f1d?source=rss-dfcee22e2d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/72d666f80f1d</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Nuatu Tseggai]]></dc:creator>
            <pubDate>Thu, 12 Jul 2018 19:29:02 GMT</pubDate>
            <atom:updated>2018-07-12T20:07:41.567Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://twitter.com/dataface">Nuatu Tseggai</a>| July 12, 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*AbwMqT39FJ-vlYuda0jPug.jpeg" /><figcaption>Practice Makes Perfect</figcaption></figure><p><em>This is part one of a multi-part blog series. In this post we’ll discuss Disaster Recovery planning when building </em><a href="https://martinfowler.com/articles/serverless.html"><em>serverless applications</em></a><em>. In future posts we’ll highlight Disaster Recovery tests, exercises and the engineering preparation necessary for success.</em></p><h3>‘Eat Your Own Dog Food’</h3><p>Nearly the entire mix of Stackery backend microservices run on AWS Lambda compute. That’s not shocking — after all — the entire purpose of our business is to build a cohesive set of tools that enable teams to build <a href="https://www.stackery.io/">production-ready serverless applications</a>. It’s only fitting that we eat our own dogfood and use serverless technologies wherever possible.</p><blockquote>Which leads to the central question this blog post is highlighting: How should a team reason about Disaster Recovery when they build software atop serverless technologies?</blockquote><p><strong>(Spoiler Alert)</strong> Serverless doesn’t equate to a free lunch! The important bits of DR revolve around establishing a cohesive plan and exercising it regularly — all of which remain important when utilizing <a href="https://www.stackery.io/resources/product-brief/">serverless infrastructure</a>. But there’s good news! Because engineers aren’t dealing with the minutia of administering a platform (patching server software, etc) they have more breathing room to focus their sights on higher level concepts such as Disaster Recovery, Security, and Technical Debt.</p><p>Before we get too far — let’s define Disaster Recovery (DR). In simple terms, it’s a documented plan that aims to minimize downtime and data loss in the event of a disaster. The term is most often used in the context of yearly audit-related exercises wherein organizations demonstrate compliance in order to meet regulatory requirements. It’s also very familiar to those who are charged with developing IT capabilities for mission-critical functions of the government.</p><p>Many of us at Stackery used to work at New Relic during a particularly explosive growth stage of the business. We were exposed to DR exercises that took months of work (from dozens of managers/engineers) to reach the objectives set by the business. That experience influenced us as we embarked on developing a DR plan for Stackery, but we still needed to work through a multitude of questions specific to our architecture.</p><p>What would happen to our product(s) if any of the following services running in AWS region XYZ experienced an outage?</p><p>(S3, RDS, Dynamo, Cognito, Lambda, Fargate, etc.)</p><ul><li>How long before we fully recover?</li><li>How much data loss would we incur?</li><li>What process would we follow to recover?</li><li>How would we communicate status and next steps internally?</li><li>How would we communicate status and next steps to customers?</li></ul><p>These questions quickly reminded us that DR planning requires direction from the business. In our case, we looked to our CEO, CTO, and VP of Engineering to set two goals:</p><ol><li>Recovery Time Objective (RTO): the length of time it would take us to swap to a second, hot production service in a separate AWS region.</li><li>Recovery Point Objective (RPO): the acceptable amount of data loss measured in time.</li></ol><p>In order to determine these goals our executives had to consider the financial impact to the business during downtime (determined by considering loss of business and damage to our reputation). Not surprisingly, the dimensions of this business decision will be unique to every business. It’s important that your executive team takes the time to understand why it’s important for them to be in charge of defining the RTO and RPO and that they are engaged in the ongoing development and execution of the DR plan. It’s a living plan and as such will require improvements as the company evolves.</p><p>Based on our experience, we developed the below outline that you may find helpful as your team develops a DR plan.</p><h3>Disaster Recovery Plan</h3><ol><li>Goals</li><li>Process</li></ol><ul><li>Initiating DR</li><li>Assigning Roles</li><li>Incident Commander</li><li>Technical Lead</li></ul><p>3. Communication</p><ul><li>Engineering Coordination</li><li>Leadership Updates</li></ul><p>4. Recovery Steps</p><p>5. Continuous Improvement</p><ul><li>TODO</li><li>Lessons Learned</li><li>Frequency</li></ul><h4>Goals:</h4><p>This section describes our RTO and RPO (see above).</p><h4>Process:</h4><p>This section describes the process to follow in the event that it becomes necessary to initiate <a href="https://aws.amazon.com/disaster-recovery/">Disaster Recovery</a>. This is the same process followed during Disaster Recovery Exercises.</p><h4>Initiating DR:</h4><p>The Disaster Recovery procedure may be initiated in the event of a major prolonged outage upon the CEO’s request. If the CEO is unavailable and cannot be reached DR can be initiated by another member of the executive team.</p><h4>Assigning Roles:</h4><p>Roles will be assigned by the executive initiating the DR process.</p><p><strong>Incident Commander (IC):</strong></p><p>The Incident Commander is responsible for coordinating the operational response and communicating status to stakeholders. The IC is responsible for designating a Technical Lead and engaging additional employees necessary for the response. During the DR process the IC will send hourly email updates to the executive team. These updates will include: current status of DR process, timeline of events since DR was initiated, requests for help or additional resources.</p><p><strong>Technical Lead (TL):</strong></p><p>The Technical Lead has primary responsibility for driving the DR process towards a successful technical resolution. The IC will solicit status information and requests for additional assistance from the TL.</p><h4>Communication:</h4><p>Communication is critical to an effective and well coordinated response. The following communication channels should be used:</p><p><strong>Engineering Coordination:</strong></p><p>The IC, TL and engineers directly involved with the response will communicate in the #disaster-recovery-XYZ slack channel. In the event that slack is unavailable the IC will initiate a Google Hangout and communicate instructions for connecting via email and cell phone.</p><p><strong>Leadership Updates:</strong></p><p>The IC will provide hourly updates to the executive team via email. See details in separate Incident Commander doc.</p><h4>Recovery Steps:</h4><p>High level steps to be performed during DR.</p><ol><li>Update Status Page</li><li>Restore Datastore(s) in prodY from latest prodX</li></ol><ul><li>DB</li><li>Authentication</li><li>Authorization</li><li>Cache</li><li>Blob Storage</li></ul><p>3. Restore backend microservices</p><ul><li>Bootstrap services with particular focus on upstream and downstream dependencies</li></ul><p>4. Swap CloudFront distribution(s)</p><p>5. Swap API endpoint(s) via DNS</p><ul><li>Update DNS records to point to prodY API endpoints</li></ul><p>6. Verify recovery is complete</p><ul><li>Redeploy stack from user account to verify service level</li></ul><p>7. Update Status Page</p><h4>Continuous Improvement:</h4><p>This section captures TODO action items and next steps, lessons learned, and the frequency in which we’ll revisit the plan and accomplish the TODO action items.</p><p>In the next post, we’ll dig into the work it takes to prepare for and perform DR tests and exercises. To learn how Stackery can make building microservices on Lambda manageable and efficient, contact our <a href="https://www.stackery.io/contact/">sales team</a> or get a <a href="https://www.stackery.io/sign-up/">free trial</a> today.</p><p><em>Originally published at </em><a href="https://www.stackery.io/blog/disaster-recovery-in-a-serverless-world-1-of-2/"><em>www.stackery.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72d666f80f1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/stacks-on-stacks/disaster-recovery-in-a-serverless-world-part-1-72d666f80f1d">Disaster Recovery in a Serverless World — Part 1</a> was originally published in <a href="https://medium.com/stacks-on-stacks">Stacks on Stacks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>