<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rafael Rodriguez on Medium]]></title>
        <description><![CDATA[Stories by Rafael Rodriguez on Medium]]></description>
        <link>https://medium.com/@rafaelrodriguez_68048?source=rss-8b010f1a6aa5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*VRWOuMf5QdOGs62G</url>
            <title>Stories by Rafael Rodriguez on Medium</title>
            <link>https://medium.com/@rafaelrodriguez_68048?source=rss-8b010f1a6aa5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:11:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rafaelrodriguez_68048/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[it’s simple with AWS: A step by step serverless fan-out architecture guide]]></title>
            <link>https://medium.com/@rafaelrodriguez_68048/its-simple-with-aws-a-step-by-step-serverless-fan-out-architecture-guide-d81f5b0e0faa?source=rss-8b010f1a6aa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d81f5b0e0faa</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[api-gateway]]></category>
            <dc:creator><![CDATA[Rafael Rodriguez]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 20:06:04 GMT</pubDate>
            <atom:updated>2020-07-06T20:06:04.232Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi everyone, this is my first post ever, and here I will start a series of posts related to AWS, where I will show how simple it can be to build scalable and highly available architectures.</p><h4>Scalable and highly available services</h4><p>If you have some experience building scalable systems, you most probably have used or heard about queues, consumers, event bus, and resource orchestration.. 😵 😵 😵 oh, that sounds complicated, and it can be! Maybe you will need to manage your Kafka cluster or RabbitMQ, and perhaps someone will say: “we need a Kubernetes cluster!”. Well, although they all work just fine, it might not be the best solution available.</p><p>If you’re looking for a quick and straightforward solution ready to scale, think twice before implementing your Kubernetes cluster and check serverless approaches.</p><p>In this guide, I will build a simple, yet robust fan-out architecture, using only AWS Services.</p><h4>Prerequisites</h4><p>For this tutorial, you will need:</p><ul><li>an AWS account</li><li>basic knowledge of AWS</li><li>basic concepts of the <a href="https://en.wikipedia.org/wiki/Fan-out_%28software%29"><em>fan-out</em></a> pattern</li></ul><h4>Fan-out</h4><p>Is a pattern where we spread messages to multiple destinations</p><h4>Why do I need to use fan-out?</h4><ul><li>If you have a use case where availability and scalabilities are requirements</li><li>If you want to decouple and make your architecture fault-tolerant</li><li>If your downstream services can and should work independently (microservices)</li></ul><h4>A “real” world use case</h4><p>Let’s think about a simple <em>Orders </em>module of a marketplace application where a user submits the order, and a bunch of things happens in our backend while he waits for a success response.</p><p>So, we can say that we:</p><ul><li>Process orders</li><li>Notify users</li><li>Prepare orders for delivery</li><li>Generate orders reports</li><li>Verify for compliance in certain countries</li></ul><p>A Monolith could efficiently execute all of these steps and might be just what you need. But if you need to scale(moving to new markets), how complex would it be, and how would you do it? Horizontally? Vertically? Do you need orchestration to scale? Kubernetes? What if one of the steps fails? Can I reprocess it? How to code that?</p><p>These are just a few questions, and it can become much more complicated, so how can we avoid it?</p><p>Fortunately, AWS has many different services to simplify our lives. It takes only a few clicks to build a highly available, scalable, and fault-tolerant fan-out architecture that will remove the burden of managing our infrastructure.</p><p><strong>A highly available and fully decoupled fan-out architecture that works with everything</strong></p><p>Following the previous use case, we can combine a set of AWS serverless services that will run independently and will scale to thousands of messages per second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*huJ9INwloZmHi6RLtC-T2A.png" /></figure><p>Here we POST our orders to the API gateway; the API gateway sends them to an SNS topic (orders topic) that takes care of distributing the event to our different queues; after that, you can process them independently in various services.</p><h4>I’ve done something similar, but I have used a lambda.</h4><p>One of the cool things about API gateway is that you can integrate it directly to AWS services, and with <a href="https://velocity.apache.org/engine/2.0/vtl-reference.html">VTL,</a> you can do all types of mappings and integrations. So, go ahead and play with it.</p><p><strong>So many pieces it doesn’t look that easy.</strong></p><p>Yes, AWS can be scary, but the most natural way of learning it is playing around, so, log in your AWS account and follow this tutorial:<br><em>(alternatively, clone the terraform project linked at the end of this post)</em></p><h4>AWS API Gateway</h4><p><em>It is a serverless service for HTTP, REST, and WebSockets APIs.</em></p><ul><li>Open: <a href="https://eu-west-1.console.aws.amazon.com/apigateway">https://eu-west-1.console.aws.amazon.com/apigateway</a></li><li>Click create API, then Rest API</li><li>Import from <a href="https://swagger.io/">Swagger</a> or <a href="https://www.openapis.org/">OpenApi</a></li><li>Paste the following code:</li></ul><pre>openapi: &quot;3.0.1&quot;<br>info:<br>  title: &quot;Fan Out Sample&quot;<br>  description: Sample API with direct integration to SNS<br>  version: &quot;0.0.1&quot;<br>paths:<br>  /market/{market}/shops/{shop_id}/oders:<br>    post:<br>      parameters:<br>        - in: path<br>          name: market<br>          description: The Market where the order should be processed<br>          required: true<br>          schema:<br>            type: string<br>            enum: [eu, us, ru]<br>        - in: path<br>          name: shop_id<br>          required: true<br>          schema:<br>            type: string<br>          description: The shop ID to whom the oder should be processed<br>      responses:<br>        200:<br>          description: &quot;200 response&quot;<br>          content:<br>            application/json:<br>              schema:<br>                $ref: &quot;#/components/schemas/Empty&quot;<br>      x-amazon-apigateway-request-validator: &quot;full&quot;<br>components:<br>  schemas:<br>    Empty:<br>      title: &quot;Empty Schema&quot;<br>      type: &quot;object&quot;<br>x-amazon-apigateway-request-validators:<br>  full:<br>    validateRequestParameters: true<br>    validateRequestBody: true</pre><p>The above steps should create a basic API without any integration</p><h4>Now let’s create the SNS topic.</h4><p><em>SNS is a high available serverless pub/sub messaging service.</em></p><ul><li>Open the SNS service in the <a href="https://eu-west-1.console.aws.amazon.com/sns/v3/home">console</a></li><li>Create a topic and name it: orders_topic</li><li>Copy the <a href="https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html">ARN</a> reference</li></ul><h4>Let’s add an <a href="https://docs.aws.amazon.com/aws-backup/latest/devguide/iam-service-roles.html">IAM role</a> to allow the API Gateway to publish in our SNS topic.</h4><p><em>IAM role is a role that our API Gateway will assume to be able to send messages to SNS.</em></p><ul><li><a href="https://console.aws.amazon.com/iam/home">Open IAM</a></li><li>Create a new policy with the <a href="https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html">ARN</a> of your topic</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Action&quot;: [<br>                &quot;sns:Publish&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Resource&quot;: [<br>                &quot;<strong>ARN_OF_YOUR_TOPIC</strong>&quot;<br>            ]<br>        }<br>    ]<br>}</pre><ul><li>Copy the policy name</li><li>Create a new AWS Role and Attach the new policy to it</li><li>Copy the role ARN</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pf1dWa6s2f8t7-yf3BGkqw.gif" /></figure><h4>Configure the <a href="https://eu-west-1.console.aws.amazon.com/apigateway/home">AWS API Gateway</a> service integration request.</h4><p><em>You can define where the messages sent to your API are going to be forwarded, and it can be an AWS service, a lambda, a VPC endpoint, or another API (HTTP).</em></p><ul><li>Go back to the API Gateway console</li><li>Select your API and click in the POST method</li><li>Click in Integration Request</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLjgXVAyOdFcuN1vJ9o3bw.png" /><figcaption>API Gateway Method details</figcaption></figure><ul><li>Enter the <strong>region</strong> of your topic</li><li>Enter <strong>AWS Service</strong>: SNS</li><li>Enter the<strong> HTTP Method</strong>: POST</li><li>Enter <strong>PATH Override</strong>: /</li><li>Enter <strong>Execution Role</strong>: The ARN of your <strong>new role</strong></li><li>Enter <strong>Content Handling</strong>: Passthrough</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ljI1wswEoqxJJPdN_Qk6OQ.png" /><figcaption>API Gateway Configuration</figcaption></figure><h4>Configure the integration request mapping template</h4><p><em>A request mapping template lets you override a request before sending it to the downstream service.</em></p><ul><li>Click in the<strong> integration request</strong></li><li>Scroll down to <strong>mapping templates</strong></li><li>Tick the <strong>never</strong> combo box</li><li>Click <strong>add mapping template</strong></li><li>write <strong>application/json</strong> and <strong>save</strong></li><li>add the following code (replace YOUR_TOPIC_ARN) and <strong>SAVE</strong></li></ul><pre>Action=Publish&amp;TopicArn=$util.urlEncode(&#39;<strong>YOUR_TOPIC_ARN</strong>&#39;)&amp;Message=$util.urlEncode($input.body)&amp;MessageAttributes.entry.1.Name=shop&amp;MessageAttributes.entry.1.Value.DataType=String&amp;MessageAttributes.entry.1.Value.StringValue=$input.params(&#39;shop_id&#39;)&amp;MessageAttributes.entry.2.Name=market&amp;MessageAttributes.entry.2.Value.DataType=String&amp;MessageAttributes.entry.2.Value.StringValue=$input.params(&#39;market&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UWQh-JMZ7VRkyROffZayww.png" /><figcaption>API Gateway Integration Request Mapping</figcaption></figure><h4>Add an HTTP Header to your integration request.</h4><p><em>You can add custom headers to your integration request using HTTP headers.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m7PGu2j1sZYC1r6mXFNxbg.png" /><figcaption>API Gateway Integration Request Header</figcaption></figure><p>At this point, the API Gateway is ready to send messages to the newly created SNS topic, and you already have a high available fan out service but still need to spread the messages, so let’s create some queues.</p><h4>Creating SQS Queues</h4><p><em>SQS is a serverless queuing service that enables you to scale and decouple microservices</em></p><ul><li><a href="https://eu-west-1.console.aws.amazon.com/sqs/home">Open SQS Console</a></li><li>Create a new Standard Queue</li><li>Copy the ARN</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5dfPBerlEJVqVH1wZrSCA.png" /><figcaption>SQS Details</figcaption></figure><h4>Subscribe your Queue</h4><p><em>Subscriptions are the way of telling the SNS topic which services are interested in the messages sent to it.</em></p><ul><li><a href="https://eu-west-1.console.aws.amazon.com/sns/v3/home">Open SNS Console</a></li><li>Select your <strong>topic</strong></li><li>Click create subscriptions</li><li>Select the <strong>protocol</strong> <strong>SQS</strong></li><li>Enter your <strong>SQS Queue ARN</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NLwNy5KOYJMOwywu0wSFJQ.png" /><figcaption>SNS Subscription</figcaption></figure><p>Create as many queues as you need, and <strong>it’s done</strong>!</p><p>Now you can start sending messages to your API Gateway, and you will be able to pool messages from your SQS queue.</p><h4>Try it out</h4><p>You can now open the API Gateway Console and test your API</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnsEcbtGEQSWfu8pQMOrOw.png" /><figcaption>AWS API Gateway Test Console</figcaption></figure><p>Now <strong>POOL</strong> your messages from the SQS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*02JQnbgRoK-Wrns6Hw3zAw.png" /><figcaption>SQS Message Detail</figcaption></figure><h4>Extra: Message Attributes</h4><p>If you explore the API Gateway / SNS integration we’ve done, you will notice the use of Message Attributes. <a href="https://docs.aws.amazon.com/sns/latest/dg/sns-message-attributes.html">Message Attributes</a> is an excellent feature of SNS where you can use simple filtering and spread only messages that matter to our subscribers.</p><p>Try cloning and provisioning the <a href="https://gitlab.com/rodriguezrps/aws-fan-out">sample,</a> then test your API setting the market as <strong>“ru”</strong> or “<strong>us” </strong>and<strong> </strong>observe the messages in your SQS queues, you will notice that the compliance queue only receives events from the market <strong>“ru”.</strong></p><h4>That looks cool, but I’m still not convinced.</h4><p>You’re probably asking yourself (and I hope you are) things like: “how does that scales?”, “What about the costs?” “Anything I should take into account?”.</p><p>The good thing about doing serverless with AWS is that they take care of everything, meaning they will scale automatically for you. But hey, your API will not be able to handle infinite requests per second, they do have some limits. This simple architecture will handle up to 10000 transactions per second (depending on the region), and that is a lot.</p><p><em>Check the following links for limits related to </em><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html"><em>API Gateway</em></a><em>, </em><a href="https://docs.aws.amazon.com/general/latest/gr/sns.html"><em>SNS</em></a><em>, and </em><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-quotas.html"><em>SQS</em></a><em>.</em></p><p>Another advantage of serverless architectures is: You don’t pay for what you don’t use, so, forget about unused infra, you don’t need to take care of those large machines that stay idle 2–3 hours per day. AWS takes care of it for you, and you only pay for what you use, making a perfect choice for early startups that don’t know exactly their loads.</p><p>And well, we didn’t write a single line of code; with just point and clicks, we were able to create a scalable fan-out architecture that you pay for what you use, and that, that is pretty awesome.</p><p>Be careful, everything looks great, and it is, but the cost can scale pretty quick, especially for SNS. Use this <a href="https://calculator.aws/#/addService">link</a> and avoid any unexpected charges. Also, set some <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/">AWS Budgets</a> for your account and be notified when you exceed them.</p><h3>Next Steps</h3><p>In this guide, we built the basics to connect AWS API Gateway with an SNS topic and an SQS queue, and we achieved an underlying fan-out architecture that scales, but we’re still missing a few things to have the right solution. So, have a look at:</p><h4>API Gateway features like:</h4><ul><li><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-setup-api-key-with-console.html">API Key</a></li><li><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-method-request-validation.html">Validation</a></li><li><a href="https://aws.amazon.com/cognito/">Cognito Integration</a></li><li><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html">Throttling</a></li><li><a href="https://aws.amazon.com/waf/">WAF Integration</a> — <strong>don’t skip this one</strong></li></ul><h4>SQS features like:</h4><ul><li><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html">Dead letter queues</a></li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html">AWS Lambdas Integration</a></li><li><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html">long and short pooling</a></li></ul><p><em>alternatively, avoid the point and click and clone a terraform source code with lambda integration </em><a href="https://gitlab.com/rodriguezrps/aws-fan-out"><em>here</em></a></p><h4>Useful links</h4><ul><li><a href="https://theburningmonk.com/2018/04/how-to-do-fan-out-and-fan-in-with-aws-lambda/">how to do fan-out and fan-in with AWS Lambda | theburningmonk.com</a></li><li><a href="https://www.jeremydaly.com/how-to-use-sns-and-sqs-to-distribute-and-throttle-events/">How To: Use SNS and SQS to Distribute and Throttle Events - Jeremy Daly</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d81f5b0e0faa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>