<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[b2w engineering -en - Medium]]></title>
        <description><![CDATA[We’re the area responsible for technology and digital services of americanas. - Medium]]></description>
        <link>https://medium.com/b2w-engineering-en?source=rss----b0bc3e71822e---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>b2w engineering -en - Medium</title>
            <link>https://medium.com/b2w-engineering-en?source=rss----b0bc3e71822e---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:11:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/b2w-engineering-en" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Delivering untested code with safety]]></title>
            <link>https://medium.com/b2w-engineering-en/delivering-untested-code-with-safety-76a7df28aebf?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/76a7df28aebf</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[e2e-testing]]></category>
            <category><![CDATA[cypress]]></category>
            <dc:creator><![CDATA[Matheus Mesquita]]></dc:creator>
            <pubDate>Wed, 30 Jun 2021 13:23:16 GMT</pubDate>
            <atom:updated>2022-05-11T22:21:12.480Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1TdsPT-h_MnGGidfrRMQ6g.png" /></figure><p><em>Great products need great technical decisions. When your tests start playing against you, what would you do?</em></p><h3>Intro</h3><p>While ago my team at B2W Digital, the biggest e-commerce from Latin America, made the decision of removing the e2e testing step from one of our project’s CI pipeline, our tests, running on a selenium-based architecture, were taking too much time to complete and was harming our development cycle.</p><p>This project is the oldest one in our team, more than 10 developers have made changes to its codebase already, and more will help in the future. It is the core part of the entire e-commerce process, helping thousands of customers to complete their shopping daily. We’re talking about the Checkout screen, a place where errors can make our company lose money.</p><p>Our decision of dropping the e2e testing was a risky one, moving the responsibility of testing to someone in the team often our PM or Team Leader, was creating a dependency on how much we can trust in the error-prone human testing it.</p><p>Without a doubt, we <a href="https://en.wikipedia.org/wiki/Human_error">lost this battle many times</a> but in compensation without the useless tests, we had a greater delivery ratio, decreasing the pressure we were and letting us to treat the other technical debts the project — which was created in a time where SPA’s were new and Angular 1 was starting to be a thing — had.</p><p>Looking back at our decisions, some questions came up.</p><ol><li>How important are e2e tests?</li><li>How can we add e2e testing to our workflow without it being a burden to us?</li><li>Which tool should we use for keeping our tests suite decoupled from the project’s framework?</li></ol><p>I’ll try to express my opinions around those three questions in this post series, I hope you enjoy it. :)</p><h3>How important are e2e tests?</h3><h3>Definitions</h3><p>Before we start diving on our discussion, lets set some meanings to our words in a way that everyone will be reading in the same language.</p><p>As said by Simon Stewart at <a href="https://testing.googleblog.com/2010/12/test-sizes.html">this blog post</a> naming test types is a hard job, there isn’t a consensus on the software developing community and, most probably, everyone reading this post has heard multiple names for the same concept.</p><ul><li><strong>End-to-end tests</strong> are the ones which verify that all parts of our applications are working <em>together</em>, from side to side. Here we test every aspect of the user interaction in our application, e.g. in a login page the user clicks on the email field, types in his email address, clicks on the password field, types in his password, clicks on login button, the page sends a request to an API and it answers with a 200 status code.</li><li><strong>UI tests</strong> are similar to <em>e2e(end-to-end)</em> tests, but they will exclusively test how our interface displays the data gathered from an API and the user interactions in the UI. Note that we don’t necessarily use the real API’s response, we can achieve the same result using <em>testing doubles(stubs and spies)</em>, e.g. in a login page, after focusing and inserting the values in our inputs, when the user clicks on the login button our spy ensures that the request was sent and our stub answer with the 200 status code.</li></ul><p>With the definition cards visible to us, let’s take a look in an old concept around tests.</p><h3>The Good Old Test Pyramid</h3><p>UI and E2E tests are commonly overlapped on the top triangle from the Test Pyramid, which is a good place to start seeking for the first question answer<em>(How important are e2e tests?)</em>. Even though it doesn’t show LITERALLY all the possible testing categories, it resumes the most common and used ones in modern software development, and that will be enough for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/0*3qQgd6rPJqifH5Sa.png" /></figure><p>As seen above, our pyramid creates the perfect analogy to how we should prioritize our tests cycle, from bottom to top. We must rather test the smallest parts of our system, the units, which are more isolated, provide quick feedback and needs fewer iterations, than the biggest one, the UI tests, which have a slower feedback cycle in addition of being hard to be crafted because of the inherent nature of user interactions being complex.</p><p>It easily shows that if we focus only on Unit Tests our development cycle becomes faster, which in the world of agile is usually preferred over quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*U9aKhPVYZoJ74gRO.png" /></figure><p>The worst scenario is when our pyramid meets Stranger Things and turns upside down. If this anti-pattern happens our application starts playing against us. E2E and UI tests have a slower execution time making our entire CI/CD process to slow down, impacting our team productivity directly. Besides, depending on how you implemented your tests, they can prove to be a nest of false positive alarms, removing all the confidence your team had on it when you started developing.</p><p><em>E2E and UI</em> tests aren’t an easy task, not as much as it is to lose the control over your testing flows and creating a gigantic technical debt in the worst place possible, the place which was supposed to bring safety to our application, our tests tasks.</p><p>This was the scenario we experienced, till the point we’ve just removed it because fixing it won’t be valuable enough in the current project condition.</p><h3>Are unit tests enough?</h3><p>So with all those downsides looks like we already got our answer about the highest aspect of testing, right? Let’s just drop it!!!</p><p><strong>Problem solved</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*uChb8GaEEYqRWu9k.gif" /></figure><p><strong>No!</strong> Every time you think unit tests are enough you’ve just made your application an unsafe place. :(</p><h3>How can we add e2e testing to our workflow without it being a burden to us?</h3><p>Let’s take a look at the problem from another point of view.</p><p>When developing software we should always aim to have <a href="https://en.wikipedia.org/wiki/Orthogonality_(programming)">orthogonal systems</a>, if we keep the parts of our application decoupled this brings safety and predictability to us. Taking this affirmation as an axiom, we should reevaluate both alternatives for the upper place of our pyramid.</p><h3>End-to-End Testing</h3><p>As mentioned in Definitions, e2e tests our application side-to-side, including the API’s responses and complex steps to reproduce a specific scenario, like those really side cases where the state created by the user was possible only after a meaningful number of different interactions in our UI.</p><p>Does anything sound weird to you? If you said yes, you are correct, having a test in a front-end project verifying a real API response is coupling your assertion to the matters of an API being working or not, and in a case where the API becomes off-line, your test job will fail with a false positive from a problem that your current project/code has nothing related with. To be clear, in this scenario your UI would break in a local test suite in a production environment, the problem is located at another system, the API.</p><p>The front-end test suit shouldn’t be dependent on the back-end environment, although it is extremely necessary toward production, for testing purposes we can stub the responses and trust that the team responsible for the API will deliver the data correctly to our front-end application and guarantee the up status.</p><p>By this perspective, E2E testing doesn’t look like a good approach for modern front-end applications since our view layer is really decoupled from our data layer, sometimes even deployed in different environments.</p><h3>UI Testing</h3><p>What really matters to a front-end application is testing if the behavior of our components are correct, if the data displayed is in the right place and if our actions are being called with user input.</p><p>UI Testing addresses exactly those points, no more, no less. Stubing our requests and having access to our state, we can imperatively enforce scenarios which a normal user session<em>(like the ones we would implement in e2e tests)</em> would take seconds or even minutes to be achieved because of the interaction with our API’s.</p><p>Which parts we should test them?</p><h3>The correct amount of tests</h3><p>It’s hard to choose which scenarios we must test, remember from <a href="https://betterexplained.com/articles/understanding-the-pareto-principle-the-8020-rule/">Pareto’s Principle</a> that 80% of our effort will be probably spent on writing the first 20% of the tests, and spending the last 20% of effort won’t be enough for the last 80% of tests, hence being counter-productive, not to mention that growing our coverage <a href="https://jasonrudolph.com/blog/2008/06/17/testing-anti-patterns-incidental-coverage/">won’t guarantee more quality</a>.</p><p>Note, I’m not telling you that 20% is enough, it depends on your project and only you with your team can decide the right amount of coverage you should target. Just keep in mind that numbers bigger than 40% are high and maybe you should reevaluate what you are testing.</p><p>For the task of choosing which parts to test I have some suggestions.</p><h3>Avoid duplication</h3><p>Although there are applications where a big number of interactions are necessary, we should see if this is a problem of our test’s design, it can happen easily with test duplication, e.g. You have two tests reproducing the login interaction before internal pages. Remember, The <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a> principle should also be applied to our tests, if you have a behavior that must be accomplished in multiple scenarios, create a separate module for it, and import on the necessary places.</p><p>Furthermore, when you duplicate your tests, you create a hard job for some future coworker or even you, when the behavior of that specific part changes, multiplying the effort to correct those tests by the times it was replicated in your test base.</p><h3>If it is hard for you, most probably, it is hard for them too</h3><p>There is no reason for a scenario where only 1% of your users get through to break your application pipeline. Sure you want to have a 100% coverage from scenarios, what better way for guaranteeing safety then testing everything? <em>Amiright?</em> No! You shouldn’t be focusing on it and potentially having incidental coverage in your project. 100% coverage should be a benefit from testing your specifications well or developing just enough software in a <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> cycle.</p><p>With that in mind, we should focus on the core features of our product, take an e-commerce application by example, the user must finalize an order.</p><h3>A new hope</h3><p>Ok, running after a hundred percent test coverage <a href="https://jasonrudolph.com/blog/2008/06/17/testing-anti-patterns-incidental-coverage/">is an anti-pattern</a>, but wouldn’t it be wonderful if something warned us of our errors before someone came to our desk or e-mail us telling something that used to work stopped working after the last feature update?</p><p>Of course It shouldn’t happen in well-designed software, but sometimes you haven’t worked in the early stages of the software you are working right now, and bad decisions made by you or your coworkers led to a status where one change in a module that was supposed to be localized, triggers an unexpected behavior on another place of the application.</p><p>Analyzing the root cause of this problem we will be faced with technical debt, which <a href="https://dev.to/caiorcferreira/using-technical-debt-as-your-next-tool-1bp6">has its ways for being “paid back”</a>, but as my experience with legacy projects showed me, even when paying it back we often see some buggy behaviors that were previously fixed to come back and haunt us because some change has just triggered it again.</p><p>Was it a problem in our fix? Or someone just had unfixed our fix?</p><p>I would guess none of that. We will be again looking at the wrong place when searching for someone to blame, look for culprits will not make our project better nor fix the bug, it will bring only new problems and can be harmful to the relationship between your team members, and to other the teams in your company.</p><h3>UI Tests as part of your development cycle</h3><blockquote><em>Keep your enemies close but your enemies closer — </em><a href="https://www.youtube.com/watch?v=DfHJDLoGInM"><em>Michael Corleone</em></a></blockquote><p>We learned from previous topics that testing every UI aspect of our application can be harmful enough to let us creating paranoias about our tests suite, also it can be counter-productive thus harming our entire development process, slowing our deliveries, unmotivating our day and stopping our <a href="https://en.wikipedia.org/wiki/Flow_(psychology)">workflow</a>. However just dropping our tests will remove any safety we had about our application, making us deliver less-quality products and decreasing the company’s trust in our product with continuous bugs popping from every deployment we make.</p><p>I’m proposing here that you should keep aiming at 100% coverage for your UI tests splitting core from non-core. Text divergence shouldn’t stop your pipeline but missing essential buttons should.</p><p>After splitting your two test suites, which I’ll discuss how in the next post, you can make the core a mandatory step to your CI cycle and the other will be run after your code was shipped to production, in a post-production step. The reason for this is quite simple, if our non-core scenarios take longer than 30min to run, for example, in the worst case we will be warned 30min after it was shipped and we will be able to rollback to the last stable deploy before any person reach us telling about the features we broke with the last feature we deployed.</p><p>After fixing the warned bug or any reported bug, remember to create a new UI test to prevent it from happening again. Don’t stop delivering features to map scenarios from your project, it should happen in a progressive way if you follow the <a href="https://medium.com/@biratkirat/step-8-the-boy-scout-rule-robert-c-martin-uncle-bob-9ac839778385">boy scout rule</a>, always leave the code you found a little better, doing it your coworkers, clients and stakeholders will thank you, and you are going to be able to deliver <em>“buggy code”</em> without causing big harms or slowing down your workflow.</p><h3>Looking Ahead</h3><p>In the next post of this series, I’m going to discuss which tool we can use to keep our business logic decoupled from the tools we are using to accomplish it in our project. Also, explain how you can do it in <a href="https://about.gitlab.com/product/continuous-integration/">Gitlab CI</a> which is an awesome open-source tool that we use in my company to create CI/CD pipelines.</p><p>I hope you enjoyed it, it was a pleasure for me to write after so long, I used to have a blog but stopped writing when an impostor syndrome hit me and made me frustrated about the stuff I used to write about. So thank you, for your time, and patience reading this article, if you have any question, criticism or wants just to say hello I’ll be happy to talk about it in the comments section.</p><p>If you want to be notified when the next post comes up you should just follow me and the <a href="http://dev.to/">dev.to</a> will do the rest. Thank you again and have a good week. :)</p><h3>References</h3><p><a href="https://en.wikipedia.org/wiki/Human_error">https://en.wikipedia.org/wiki/Human_error</a><br><a href="https://martinfowler.com/articles/practical-test-pyramid.html#UiTests">https://martinfowler.com/articles/practical-test-pyramid.html#UiTests</a><br><a href="https://testing.googleblog.com/2010/12/test-sizes.html">https://testing.googleblog.com/2010/12/test-sizes.html</a><br><a href="https://dzone.com/articles/should-you-aim-for-100-percent-test-coverage">https://dzone.com/articles/should-you-aim-for-100-percent-test-coverage</a><br><a href="https://betterexplained.com/articles/understanding-the-pareto-principle-the-8020-rule/">https://betterexplained.com/articles/understanding-the-pareto-principle-the-8020-rule/</a><br><a href="https://jasonrudolph.com/blog/2008/06/17/testing-anti-patterns-incidental-coverage/">https://jasonrudolph.com/blog/2008/06/17/testing-anti-patterns-incidental-coverage/</a><br><a href="https://en.wikipedia.org/wiki/Test-driven_development">https://en.wikipedia.org/wiki/Test-driven_development</a><br><a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">https://en.wikipedia.org/wiki/Don&#39;t_repeat_yourself</a><br><a href="https://dev.to/caiorcferreira/using-technical-debt-as-your-next-tool-1bp6">https://dev.to/caiorcferreira/using-technical-debt-as-your-next-tool-1bp6</a><br><a href="https://medium.com/@biratkirat/step-8-the-boy-scout-rule-robert-c-martin-uncle-bob-9ac839778385">https://medium.com/@biratkirat/step-8-the-boy-scout-rule-robert-c-martin-uncle-bob-9ac839778385</a><br><a href="https://www.youtube.com/watch?v=DfHJDLoGInM">https://www.youtube.com/watch?v=DfHJDLoGInM</a><br><a href="https://about.gitlab.com/product/continuous-integration/">https://about.gitlab.com/product/continuous-integration/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76a7df28aebf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/delivering-untested-code-with-safety-76a7df28aebf">Delivering untested code with safety</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The reasoning behind the Expectation-Maximization (EM) Algorithm]]></title>
            <link>https://medium.com/b2w-engineering-en/the-reasoning-behind-the-expectation-maximization-em-algorithm-4a773428b3fc?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/4a773428b3fc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Alexandre Henrique]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 13:37:58 GMT</pubDate>
            <atom:updated>2021-10-15T13:38:21.190Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sYx-GdJnhg4pAulbFv0cwA.png" /></figure><h3>The Expectation-Maximization (EM) Algorithm</h3><h4>Understanding the motivations and how the EM Algorithm works</h4><p>Clustering algorithms play an important role in understanding data. Grasp relationships and detect similarities among thousands of records in a dataset is a burdensome task. Fortunately, numerous algorithms come to aid in these matters. Generally, such algorithms make different assumptions about the underlying probabilistic generative process of the data.</p><p>Here, we are gonna dive into the mathematical aspects of the Expectation-Maximization (EM) algorithm. By visiting the literature, you will discover that this technique is employed in Density Estimation problems.</p><p>Such problems aim to represent data in a compact form using a statistical distribution, e.g., Gaussian, Beta, or Gamma. You can think of those problems as a clustering task but from a probabilistic point of view. This is what makes the EM algorithm a probabilistic generative model.</p><blockquote>By probabilistic generative model, think of it as a model that instead of doing “hard” assignment providing a cluster for each data point (e.g., <a href="https://medium.com/@alexandre.hsd/everything-you-need-to-know-about-clustering-with-k-means-722f743ef1c4">K-Means</a>). The model will provide the probability that a given data point belongs to each cluster. This is called “soft” assignment.</blockquote><p>However, you may expect that a single Gaussian with its mean and variance cannot map thousands of instances in a dataset into a set of <em>K</em> clusters accurately, nor could any other distribution. As you can see in the plot below, we would need at least 3 clusters to decently segment the data. In this context, to start our discussion, we are gonna talk about <strong>Mixture Models </strong>(<strong>MM</strong>) — more specifically, <strong>Gaussian Mixture Models </strong>(<strong>GMM</strong>).</p><figure><img alt="Plot of three different clusters on a cartesian-plane with different centers and unit variance" src="https://cdn-images-1.medium.com/max/1024/1*3l-fmy7jpt7d9JFqdwY_jg.png" /><figcaption>Dataset with 3 clusters</figcaption></figure><h3>(Gaussian) Mixture Models</h3><p>To build a density estimator model, we cannot rely on a simple distribution. <strong>Mixture models</strong> try to tackle this limitation by combining a set of distributions to create a convex space where we can search for the optimal parameters for such distributions using <strong>Maximum Likelihood Estimation</strong> (<strong>MLE</strong>).</p><h4>Mixture Models</h4><p>A Mixture Model is expressed by the following equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*uQIZOwaGVob1_A7A51cPEQ.png" /><figcaption>Mixture Model</figcaption></figure><p>Where <strong><em>K</em></strong> is the number of mixture components (clusters), <em>πⱼ</em>’s are the mixture weights, and <em>pⱼ</em>(<em>x</em>)’s<em> </em>are members of a family of distributions (Gaussian, Poisson, Bernoulli, etc).</p><p>Consequently, a GMM is a Mixture Model where the <em>pⱼ</em>(<em>x</em>) is a finite combination of Gaussian Distributions. Therefore, a GMM can be precisely defined by the following set of equations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*6sMOgTwjAPVk3OiULfI7Rw.png" /><figcaption>Gaussian Mixture Model</figcaption></figure><p>Where <strong><em>θ</em></strong> is the collection of all the parameters of the model (mixture weights, means, and covariance matrices):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*gFvqALm6ef4w1SVYtK_ZIA.png" /><figcaption>Parameter Set of a GMM</figcaption></figure><p>With this convex combination of distributions, we gain a powerful tool to represent multimodal data. The following plot shows what a GMM derived from 3 mixture components looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j9d-ETANKuvNa4QZKWOrLA.png" /><figcaption>GMM derived from 3 mixture components</figcaption></figure><p>As a consequence, for each data point, <strong><em>x</em></strong> (in red), we can compute the probability that it belongs to each component (make a “soft” assignment). This quantity is called “responsibility”, and it will be formally defined later on.</p><p>Lastly, recall that equation (3) gives the likelihood of a point <strong><em>x</em></strong> in a GMM. In this particular case, it can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*RtTAM1t2PV-Fg-jsj56oeg.png" /><figcaption>Example of Gaussian Mixture Model with 3 mixture components</figcaption></figure><h3>Finding the optimal parameters of a GMM</h3><p>Given a dataset <em>X</em> of <em>n</em> data points i.i.d. with an unknown distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*YUDj8HN4BjGb8mPe4n3cQA.png" /><figcaption>Dataset of <strong><em>n</em></strong> d-dimensional data points</figcaption></figure><p>The Goal of the GMM is to represent the distribution of the data as accurately as possible using a linear combination of Gaussian Distributions. Furthermore, estimating the parameters of a GMM is driving by the goal of <strong>maximizing the GMM likelihood with respect to the parameter set θ</strong> over the dataset <em>X</em>.</p><p>Since our samples are i.i.d., the maximum likelihood estimator over <em>X </em>can be expressed as the product of the individual likelihoods. To simplify the equations, we are gonna directly apply the logarithm to the MLE function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*WCDvRYqgjJkCh4CcAl5oHA.png" /><figcaption>Log-likelihood of the data over θ</figcaption></figure><p>From equation (3), we know that <em>p(</em><strong><em>xᵢ</em></strong><em>|</em><strong><em>θ</em></strong><em>) </em>is a linear combination of Gaussian distributions. Therefore, plugging (3) into (5), we have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*652NRSGD3o5bqmys7ckgwQ.png" /><figcaption>Explicit log-likelihood of the data over θ</figcaption></figure><p>Equation (6) is not tractable in the setting of GMMs. In other words, there is no closed-form solution to finding an optimal parameter set <strong><em>θ</em></strong>, meaning that we won’t get an analytical solution by just taking the derivative of (6) with respect to <strong><em>θ</em></strong> and setting it to 0. The set of equations (7-9) outline how we would evaluate it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*pwAuy4dvMClfXPCQwM8s1g.png" /><figcaption>Set of equations defined to optimize log-likelihood of the data over θ</figcaption></figure><p>Observe that the computation of each parameter from<em> </em><strong><em>θ</em></strong> (<strong>μ</strong>’s, <strong>Σ</strong>’s, <strong>π</strong>’s) depends on the other parameters in a complex way. Consequently, there is no closed-form solution to these equations. You’ll see this clearer soon enough.</p><p>To solve those equations, we can use the strategy of optimizing some parameters while keeping the others fixed. For this purpose, we are gonna use the Expectation-Maximization algorithm.</p><h3>The EM Algorithm</h3><p>Before dive into the iterative process of finding the optimal parameters of a GMM — described by the EM algorithm. Let’s define a crucial quantity that will allow us to write the equations in a compact form.</p><h4>Responsibilities (the E step)</h4><p>As seen before, the GMM is built over the composition of other simple Gaussian distributions. The amount of contribution of each Gaussian over the quantity outputted by the GMM is called <strong>Responsibility</strong>.</p><p>The responsibility <em>rᵢⱼ</em> can be seen as the posterior probability that point <em>xᵢ </em>was generated from cluster <em>j</em>. Put in simple words, <strong>the responsibility <em>rᵢⱼ </em>is the probability that the <em>j</em>th mixture component generated point <em>xᵢ</em></strong><em>. </em>Mathematically, the responsibilities can be computed as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*F7LFIbQyGjXe_mwiLmYYLg.png" /><figcaption>Responsibility</figcaption></figure><p><em>Note that to compute each responsibility, we need previous knowledge about </em><strong><em>θ</em></strong><em>. </em>Thus, for each data point <em>xᵢ</em>, we’ll have a vector of length <em>K</em>, where each position indicates the probability that <em>xᵢ </em>belongs to each mixture component.</p><p>Moreover, from (10), we can define the <strong>total responsibility</strong> of the <em>j</em>th mixture component for the entire dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*zHKMOl-pYzkJ7hD7NXxKyQ.png" /><figcaption>Responsibility of the <strong>j</strong>th component over the entire dataset</figcaption></figure><h4>Estimating the Parameters of a GMM (the M step)</h4><p>We will not walk through the proofs on how to learn the optimal parameters of a GMM in order to maximize the log-likelihood of the data w.r.t <strong><em>θ</em></strong> since they are long and hard enough to have a post for their own.</p><p>Formally, there’s a consistent way of updating the individual parameters of a GMM given prior (initialized at random) parameters <strong>μ</strong>’s, <strong>Σ</strong>’s, and <strong>π</strong>’s. This approach works by updating some parameters while keeping the others fixed. Equations (12-14) summarizes the equations used for that purpose:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*HqaxQQCBHhU4Ww5M6rcoPA.png" /><figcaption>Update of the parameters of the GMM</figcaption></figure><p><strong>Note that the update of μ, Σ, and π, all depend on the responsibilities (<em>rᵢⱼ</em>), which by its turn, depends on μ, Σ, and π. </strong>That’s why there&#39;s not a closed-form solution to equations (7–9).</p><blockquote>Furthermore, equations (12–14) do not aim to precisely maximize over <strong>θ </strong>the actual log-likelihood, given in equation (6). Instead, they maximize a proxy function of the log-likelihood over <strong>θ</strong>, namely, the <strong>expected log-likelihood</strong>, which can be derived from equation (6) using <strong>Jensen’s inequality</strong>. This proxy function is given by:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*iyAkKtgsNFgPqz_xdezP9g.png" /><figcaption>The expected complete log-likelihood</figcaption></figure><h4>Iterative process</h4><p>What we’ve seen until now, points to an iterative process that can help us achieve the task of finding a (local) optimal set of parameters <strong><em>θ</em></strong> — the EM algorithm.</p><blockquote>The EM algorithm has that name because it consists of an iterative process that alternates between two steps. The first step is to compute the responsibilities (E step) of each mixture component for each data point using the current parameters (<strong>μ</strong>’s, <strong>Σ</strong>’s, and <strong>π</strong>’s). The second step consists of updating the parameters (M step) in order to maximize the expected log-likelihood given by equation (15).</blockquote><p>The E and M steps are repeated until there is no significant progress in the proxy function of the log-likelihood computed after the M step. In summary, it can be described in 4 steps:</p><pre>1. Randomly initialize μ&#39;s, Σ&#39;s, and π&#39;s;<br>2. E-Step:<br>    Compute the responsibilities for every data point <em>xᵢ </em>using μ, Σ, and π, and the equation (10):</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*F7LFIbQyGjXe_mwiLmYYLg.png" /><figcaption>Responsibility</figcaption></figure><pre>3. M-Step:<br>    Update the parameters μ, Σ, and π (plugging the responsibilities evaluated in the E-Step) using the set of equations (12-14):</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*HqaxQQCBHhU4Ww5M6rcoPA.png" /><figcaption>Update of the parameters of the GMM</figcaption></figure><pre>4. Repeat steps 2 and 3, until there is no significant change in the proxy function of the log-likelihood given by equation (15):</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*iyAkKtgsNFgPqz_xdezP9g.png" /><figcaption>The expected complete log-likelihood</figcaption></figure><p><strong><em>This procedure monotonically increases the expected log-likelihood.</em></strong></p><p>The EM algorithm is designed to work with high-dimensional data. However, for the sake of visualization, I wrote the code of the EM algorithm from scratch, ran it against a randomly generated dataset of one-dimensional data, and randomly initialized the parameters of the GMM. As expected, I got the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iRTc264XIPghpllPlh7JUg.gif" /><figcaption>EM algorithm running example</figcaption></figure><p>As you can see, the algorithm took 41 iterations to fully converge. In the plot on the right, we can see the expected log-likelihood slowly increasing until it finally stops. You can see the code of this example and the EM algorithm on my <a href="https://github.com/alexandrehsd/em-algorithm">GitHub</a>.</p><h3>Final considerations</h3><p>The EM algorithm is very sensitive to initialization. What some people recommend is to run <a href="https://medium.com/@alexandre.hsd/everything-you-need-to-know-about-clustering-with-k-means-722f743ef1c4">K-Means</a> (because it has a lower computational cost) and use the output centers as the initialization means of the mixture components. By doing that, you substantially accelerate the convergence of the EM algorithm. I would add that it is also easier to find an appropriate number of clusters by running <a href="https://medium.com/@alexandre.hsd/everything-you-need-to-know-about-clustering-with-k-means-722f743ef1c4">K-Means</a>.</p><p>Nevertheless, the EM algorithm is considered to be better than <a href="https://medium.com/@alexandre.hsd/everything-you-need-to-know-about-clustering-with-k-means-722f743ef1c4">K-Means</a> because it provides additional information about the data, namely, the dispersion (variance) of the cluster, not only its centers.</p><p>Moreover, there is an easy way of adapting the original setup of the EM algorithm to deal with user-item collaborative filtering for recommender systems.</p><h3>References</h3><ol><li><a href="https://www.edx.org/course/machine-learning-with-python-from-linear-models-to">Machine Learning with Python: from Linear Models to Deep Learning, a course by MITx</a>.</li><li><a href="https://mml-book.github.io/">Mathematics for Machine Learning, by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong.</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a773428b3fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/the-reasoning-behind-the-expectation-maximization-em-algorithm-4a773428b3fc">The reasoning behind the Expectation-Maximization (EM) Algorithm</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design & Technology]]></title>
            <link>https://medium.com/b2w-engineering-en/design-technology-d7dce4a142ef?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/d7dce4a142ef</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[b2w-digital]]></category>
            <dc:creator><![CDATA[Rodrigo Padin]]></dc:creator>
            <pubDate>Wed, 09 Jun 2021 14:05:19 GMT</pubDate>
            <atom:updated>2021-06-18T13:33:20.107Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*igQVGVnZRGOnkge7GiPfHw.png" /></figure><p><em>Design and technology are complementary and should work in tandem in companies.</em></p><p>Design involves working with values ​​and expectations in many different ways, designers have the responsibility to deliver great experiences. To do this, they need to see from a wide variety of perspectives, connect with people and understand what and how they feel.</p><p>The design team doesn’t just create visuals. Apart from understanding people, it also understands business needs, explores endless detailed options, tests, measures and revisits the beginning of the process for a continuous evolution.</p><p>We are talking about design as an area, but there are plenty of specializations, such as product design, UX / UI design, web design, graphic design, art direction etc.</p><p>Technology evolves all the techniques, methods, processes and studies on how to apply discoveries and tools from different scientific strands to achieve human goals. Technological developments have an intrinsic relationship with business interests and the socio-cultural needs in which they occur.</p><p>Regardless of the origin, it is essential to have a representative number of people so that the market allocates resources to the technological innovation of its products and services. For example, the creation of a system that allows us to have medical appointments without leaving home due to a Covid pandemic, or a new way of using resources for a factory so that it can reduce pollution, or even a high resolution mobile phone camera for a group of people influenced by advertising.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_WgPKf8YL8oO7d3hylt_w.png" /><figcaption>Design + Technology</figcaption></figure><p>Although for many people they are distant and distinct disciplines, they share the same role: transforming ideas into products and services that we use in our every day lives.</p><p>At B2W we are experiencing a disruption of the traditional model of teams formed by professionals with specific skills, to teams with diverse cultural and capabilities working together. This engagement between design and technology creates a connected community that speaks the same language and achieves positive and innovative results.</p><h3>Introduction</h3><p>In 2017, our data platform bee was launched, fully focused on cloud and user experience.</p><p>Developed by talented and extremely technical professionals, our structure empowers developers, data analysts, data scientists, data engineers and business analysts to optimize processes and expand possibilities. Our mission is to organize all company information, in a way that it is accessible and useful to people and systems.</p><blockquote>The name bee is a reference to the collaborative and organized work of bees and their important role in pollinating nature, in our case, this means democratizing data.</blockquote><p>Our ambition is to turn B2W into a global reference in engineering and our culture of data took this first step.One detail we were missing at the start of this journey was diversity — this is where my story at the company begins.</p><h3>Diversity and collaboration</h3><p>2 years ago I embraced the opportunity to be part of the leading digital company in Latin America, leaving behind the world of freelance designer projects. To add to this lifestyle change, I arrived into a strictly technical team quite by accident. I had no idea of which role I would occupy, even my colleagues didn’t understand the need for me to be there.</p><p>I thought, “well, I gotta start somehow, I just can’t stand still” and started looking for something to do. I got all the visual material I had found, put everything in order, created brand guidelines, visual identity styles, templates of all kinds.</p><p>In a short time, I was integrated with our team and everything related to communication. I was involved in illustrations, ads, videos, slides, documentation, events, etc. And with that, I was absorbing things and diving into the environment. I was the only designer on the team, but I never worked alone.</p><p>Product professionals also started to be hired and at a certain point this work brought consistency to the brand, not only with visual templates and language standards, but with communication plans that showcased us internally and externally, showing the relevance of our activities and products, attracting new users to the data platform and encouraging the use of data in decision making.</p><p>Design and technology were working in parallel but when did these disciplines get together? The example is our data discovery tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4W55qFNU4QI4zQ1t8P7Mg.png" /><figcaption>Creation process of product or feature</figcaption></figure><h4>Discovery</h4><p>We realised that most people didn’t know about the existence of a lot of the data already available to everyone in the company, and also the lack of access to relevant data for certain teams. This resulted in duplicate databases, differences between data values ​​in similar tables, inconsistent reports, meaning that our dream of a culture was delayed.</p><h4>Understanding and empathy</h4><p>From time to time we visit old ideas, discontinued projects and lost stories in our backlog to maybe find something that could be useful to create a solution to problems. Then, rummaging around our drawers, we found an initial proposal for a data discovery project called Explorer.</p><p>This is the time to empathize and understand people’s needs through research, observations, interviews, benchmark studies and any other way of delving into the subject and defining the objectives.</p><h4>PoC (Proof of Concept)</h4><p>We took the proposal to the next stage, where we considered if the hypotheses were technologically viable for development and if we had the necessary and available inputs to put them into practice. We studied and tested several technologies and resources in order to reach a starting architecture.</p><p>This proof of concept allowed us to collect internal and external feedback, to run tests and allowed the correction of errors and the implementation of improvements.</p><h4>Prototyping</h4><p>The first prototype of the Explorer was made with pen and paper, afterwards we evolved it so we could run tests with users, analyze their journey and collect feedback.</p><p>Similar to a proof of concept, prototypes are essential to identify if your interface is in accordance with what you want, and to align expectations with the view of the final product.</p><p>They allow closer examination of the product’s characteristics, reduce risk and wastage with tests or interactions. Prototyping provides a reference for developers in programming because it is a visual and functional tool, as well as being a reference for stakeholders.</p><p>Prototypes can have several fidelity levels: from developed with only pen and paper to high fidelity and complexity with lines of code simulating behaviors, interactions and functionalities.</p><h4>MVP (Minimum Viable Product)</h4><p>With the prototype validated, the development team started to actually build the product. It was an initial version of a product with a small number of features, used to test with a limited number of people that are part of our target audience to find out if the solutions really work and provide value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H8ReZSfKcZ5f4bdNPoiJzw.jpeg" /><figcaption>Explorer’s screen details</figcaption></figure><h4>Release and continuous delivery</h4><p>Based on the test results and feedback, we made all the necessary adjustments and finally the product’s official release has come, but our job didn’t end up here — a good solution is constantly evolving, isn’t it?</p><p>Today we divide our time in creating and implementing new features, improvements and bug fixes. And of course, we are always open to new ideas.</p><h3>Conclusion</h3><p>A new product comes out aiming to solve a problem and to relieve people’s pain, the demands for its creation can appear at any time through insights, requests from the client, changes in the legislation or among other circumstances.</p><p>From different points of view and experiences, the process of creating and developing a new technology product, with a multidisciplinary team, makes it more agile, avoids rework and adds greater value to customers by having well-targeted solutions to achieve set goals.</p><p>Today, in addition to a consolidated platform, we organize events, offer training and develop tools to empower people and boost the data-driven culture in the company — our internal community has grown to more than 2,000 members.</p><p>If you are looking for a development opportunity, working with innovation in a high-impact business, visit the <a href="https://carreiras.b2w.io/">B2W Careers portal</a>! In it, you can access all available positions. Come join our team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7dce4a142ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/design-technology-d7dce4a142ef">Design &amp; Technology</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integration test with Go,  Wiremock and Mongodb using dockertest]]></title>
            <link>https://medium.com/b2w-engineering-en/integration-test-with-go-wiremock-and-mongodb-using-dockertest-130bf7d3be5c?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/130bf7d3be5c</guid>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[wiremock]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Mahmoud aljadan]]></dc:creator>
            <pubDate>Wed, 19 May 2021 14:38:50 GMT</pubDate>
            <atom:updated>2021-06-18T13:34:00.125Z</atom:updated>
            <content:encoded><![CDATA[<h3>Integration test with Go, Wiremock and Mongodb using dockertest</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fsz4c3os_gIBV6q5pt9l0A.png" /></figure><p>In most of the production ready APIs you need to have integration tests which is going to test everything related to API logic, from validations, calculation up to the response format.</p><p>But most of the APIs have different external dependencies such as ( Messaging System, Other API call, Monitoring tools, Tracing tools, etc…) which make integration test much harder.</p><p>Luckily we can use docker to mock all of these external dependencies by spinning up a container for each dependency and that is done by either writing a docker-compose file to orchestrate these containers or by using a docker library where we are going to write a function to spin up these containers before we run our tests.</p><p>Here we are going to see how to use <a href="https://github.com/ory/dockertest"><strong>dockertest</strong></a><strong> </strong>to write our integration tests.</p><p>In our example i have created a simple API which has two dependencies, one will access mongodb and the other will make a rest call to another API, In order to write tests for these endpoints in isolation of its dependencies, we are going to mock the API call using Wiremock container and for the database we will run a mongodb test container.</p><p>Let get into the code now.</p><p>End points:</p><ul><li>GET /resource-status ( static response to check if the API is working)</li><li>Get /planets/:id ( call Star War API )</li><li>Get /people/:id ( get document from mongodb)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96cf6ad40e0cd8c2c294a000a663eb30/href">https://medium.com/media/96cf6ad40e0cd8c2c294a000a663eb30/href</a></iframe><p>The idea here is to start up mongodb and Wiremock and our API and make a normal http call to our API to test it.</p><p>This will be done by writing a <strong>TestMain</strong> <strong>func</strong> which is going to setup and spin up our containers before running our tests.</p><p>It is important to mention here that in order for the containers to communicate with each others they require to be on the same <strong>docker network</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16cb02d0f644aaddf6113a50fe99a66c/href">https://medium.com/media/16cb02d0f644aaddf6113a50fe99a66c/href</a></iframe><p>By using the containers names and their respective internal ports, we will be able to call mongodb or wiremock from our API.</p><p>To spin up a Wiremock container, we need to mount a our stubs inside the container and use its exposed port to call it in the Retry method to check that Wiremock is up.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/729cd025adcc1fdc33493c7c494abafa/href">https://medium.com/media/729cd025adcc1fdc33493c7c494abafa/href</a></iframe><p>We do the same thing in for mongodb container, we also mount our init_db.js script inside the container to initialize our test database and we use the exposed port to connect to mongodb inside the Retry method to check that its up and running.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0abc64432e842b1bc91f2d93fd0e942c/href">https://medium.com/media/0abc64432e842b1bc91f2d93fd0e942c/href</a></iframe><p>And finally we spin up our API container using a Dockerfile which will build our application and run it. we have also attached the logger to see our API logs (for debugging purpose).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9942c7ce3a3b8b386650cbdbac8df01e/href">https://medium.com/media/9942c7ce3a3b8b386650cbdbac8df01e/href</a></iframe><p>Now we are ready to write our tests by making a normal http calls to our API endpoints.</p><p>starting the first test by calling /planets/:id</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/535a9edd69f82a098dabf62969139efe/href">https://medium.com/media/535a9edd69f82a098dabf62969139efe/href</a></iframe><p>And the second test by calling /people/:id</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2c82f29e7320525e35775f35137d716/href">https://medium.com/media/a2c82f29e7320525e35775f35137d716/href</a></iframe><p>We have also added <strong>build tags </strong>to avoid running integrations tests while running unit tests, this will added to all integration test files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c25669c3b88fa70756b0910e43d69e61/href">https://medium.com/media/c25669c3b88fa70756b0910e43d69e61/href</a></iframe><p>to run the integration test you can run :</p><p>go test -tags integration</p><p>I was not able to show all the code here, please check out the repository on github.</p><ul><li><a href="https://github.com/mahjadan/go-integration-test.git">https://github.com/mahjadan/go-integration-test.git</a></li></ul><p>If you are looking for a development opportunity, working with innovation in a high-impact business, visit the <a href="https://carreiras.b2w.io/">B2W Careers portal</a>! In it, you can access all available positions. Come join our team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=130bf7d3be5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/integration-test-with-go-wiremock-and-mongodb-using-dockertest-130bf7d3be5c">Integration test with Go,  Wiremock and Mongodb using dockertest</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lambdas with X-Ray]]></title>
            <link>https://medium.com/b2w-engineering-en/lambdas-with-x-ray-27403463d97c?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/27403463d97c</guid>
            <category><![CDATA[traceability]]></category>
            <category><![CDATA[tracking]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws-x-ray]]></category>
            <dc:creator><![CDATA[Matheus Castro]]></dc:creator>
            <pubDate>Wed, 25 Nov 2020 13:12:52 GMT</pubDate>
            <atom:updated>2021-06-18T13:34:45.364Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How and when we monitoring our functions? Not only CloudWatch monitoring a lambda.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBX3IzTzXAUzHxohH6Cmhg.png" /></figure><p>Behind the many microservices that we have, the lambdas are too part of our technical structure. Lots of them do simple operations but are fundamental to the entire business ecosystem.</p><p>Lambdas are born and die much more easily because their main function is to execute some coded logic. If something goes wrong when running and there is no part focused on it, it just dies.</p><p>With this, it is important to be concerned with the life cycle of your function, knowing if they did what had to do if something went wrong or it didn’t even perform it.</p><h3>Observability, Monitoring, and Analysis</h3><p>In the pyramid of power, observability, monitoring, and analysis combine what we call <em>“actionable intelligence”</em></p><p>The quote “If you are observable, I can understand you” can be read in another way “If you wrote something that I can see, so I can find it” is a common fact for many developers who worked with some power of APM. These tools have SDK, agents, and other things that can observe your applications, monitoring the pieces of information, and available your data for an individual analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*McowxC0WuQj3Kgw51IkCqg.png" /><figcaption><a href="https://www.instana.com/media/Observability-Monitoring-Analysis-Pyramid.png">The Pyramid of Power</a></figcaption></figure><p>Okay! Relationship with all these facts of the pyramid is important to go along huge applications, but why not with functions?</p><h3>Lambda and your tracking</h3><blockquote>Throws first stone whos never logging each step in his lambda for understanding what’s happening.</blockquote><p>The good thing about the lambda is that simple to use, and the bad thing is what’s going on inside? Looking to the CloudWatch and see a simple log START, END, REPORT, is the reason for a smile on a face and also a question your head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSwNLrRE-Cn_IUE5b_6myg.png" /><figcaption>Wow! So nice, but what did he do?</figcaption></figure><p>Using Logger’s for tracking is not considered a bad practice but has a different purpose. Before these moderns tools became famous, everything around the applications was logging and many legacy projects yet using.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1eTb06EsvYURj5bjTdnig.png" /><figcaption>Where is ally? (Keep calm is just an example)</figcaption></figure><p>One of the problems that excessive logging can cause is, in the midst of so much, losing some really relevant information. For example, within many infrastructure logs, some logs disappear about a failure in your logic or the implementation of the business rule. Together or even after tracking the HTTP call, we can lose significant information inside the response body. Believes or not, exists a balance of more and less logging. If you want to know more about this, <a href="https://www.scalyr.com/blog/the-10-commandments-of-logging/">read this article from Dave McAllister</a>.</p><h3>AWS X-Ray</h3><p>According to AWS, X-Ray is a service to helps developers to analyze and debug production, distributed applications, such as those built using a microservices architecture. They have integrations with almost all AWS Services, beyond the Lambda, likes API Gateway, EC2, Beanstalk, SNS, SQS, etc. Works with many languages like Go, Java, .Net, Python, Node, Ruby.</p><p><strong>The point here is not talking about how X-Ray works, but why and when we used it.</strong></p><blockquote>Before we go on, maybe you need some concepts about it.</blockquote><h3>Segments and Subsegments</h3><p><strong>Segments</strong> are the JSON representation that a resource request inside your code. In other words, it is like a medical report with all information about what happens inside your piece of code. This report has a name, source, execution time, details about what is requested, and if errors or failures occur.</p><p>When a function realizes some HTTP request, it is saved what sends it (request, almost all) and what received (response, almost all too).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*rBct_UDjPHxNxEy5bRBblA.png" /><figcaption><a href="https://docs.aws.amazon.com/xray/latest/devguide/images/scorekeep-PUTrules-segment-overview.png">Segments</a></figcaption></figure><p><strong>Subsegments</strong> are internal divisions of the Segment. The famous divide to conquering. When having these divisions, we have a greater granularity about the executions. An HTTP call can generate many subsegments but generally create two subsegments: request and response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*HKx-qcXjZqT1LUir2OuZ8w.png" /><figcaption>Subsegments</figcaption></figure><h3>Analysis of an Event Architecture</h3><blockquote>Exemplifying our use in a real case of a project</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BB_W-IKK4AZcoIMwrGj7vg.png" /><figcaption>Event-Oriented Architecture</figcaption></figure><p>In the image above, we have an event-oriented architecture, where has many triggers from many sources like SQS, SNS, Kinesis Streams, and DynamoDB Streams.</p><p>Each lambda is responsible for doing your part of the job, having your internal/external dependencies, but how knows when there is a bottleneck on the flow? A great point is CloudWatch Alarms with monitoring and notifying. However, how to analyse, debugging, or understand what truly happens without a new line of code?</p><h4>Service Map</h4><p>The Service Map is a map with event tracking from services. Duh!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AU0CXcEnUUoziVWfQ_BI0w.png" /><figcaption>Service Map — Resize nodes by Traffic</figcaption></figure><p>We can see has a map that shows what the events from and where they go. Look up and see the big and little circles. The Service Map allows resizing circles for traffic or health. Has some too yellows lines around the circle that indicate how many errors happened when trying to call that service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nK0_6oOaIC6H-sgqUw5TYw.png" /><figcaption>Service Map — Details of part of the flow</figcaption></figure><p>Selecting a little part of the flow, we can identify that 61% of calls that Lambda did an API have a failure.</p><h4>Tracking</h4><p>Analyzes some trackings of this specific Lambda, we can see they have HTTP errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSRU4JzfUlFEKE-xpxME9w.png" /><figcaption>Segments with errors</figcaption></figure><p>One of the steps of this function is to call an API and identify some HTTP error returns. The way with how was coded, this function couldn’t continue the flow if have returned some errors. Talking to the responsible team, we discovered an instability in their API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vDs3vqQFMATo5r_Y1pTNLQ.png" /><figcaption>Segments without errors</figcaption></figure><p>After the fixes were possible to see that flow came back to your normal state and calling your other dependencies.</p><h4>Annotation and Metadata</h4><p>An interesting feature is adding more information in your segment with annotation or metadata. <strong>Annotation</strong> is an indexed key-value and can be used in search trackings with 50 annotations limitation. <strong>Metadata</strong> is key-value too. However, other value types can be used as an object or a list. It’s not indexed, so it’s not searchable.</p><p>We use annotation with easy information that can be identified as a brand, product id, and other important things. The metadata is more used for body information from request or response.</p><h4>Context Propagation</h4><p>This feature is cool but still falls short in the way how they implemented. Context propagation is fundamental for continuous tracking. What does that mean? Keeps the same identification from the first tracking until the last tracking, even when he passes in other services, like SQS, SNS, and API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*yNaWSUS3cJHH4mGKGUOn-w.png" /><figcaption><a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html">SQS Context Propagation</a></figcaption></figure><p>The X-Ray uses the Tracing Header, which informs the identification in the header of the requests, but where’s the bad thing?<br>This step is not 100% automatically and needs to codify to propagate context. The AWS SDK can resolve to send with the Tracing Header but not from receiving.</p><h3>Why X-Ray?</h3><p>Today have many other services that can be the same functions as X-Ray, like AppDynamics, DataDog, and NewRelic. The answer to this question is more temporal and relates to the time than a simple feature comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AzBQ-Ub-A57W1tCLIWNaw.png" /><figcaption>I swear this was a checkbox</figcaption></figure><p>When we need to understand better what happened inside of our functions, the X-Ray purpose an easy integration like activate tracing with options in the Lambda panel and with less modification in our code.</p><h3>Billing</h3><p>At that time, one of the things that were not good for us is not the price but the way how to identify which lambda generates what billing. The X-ray doesn’t separate the tracks with tags and made it difficult for us to understand. We have many teams with large projects then came to be more complex to know what project was responsible for each X-Ray billings.</p><h3>Conclusion</h3><p>For us, the X-Ray was a great solution and friendly that helped identify many problems during architecting and the evolutions of projects.<br>Today’s have many other services, but in the past was the way that we found for helping us to understand and monitor our functions.</p><p>If you are looking for a development opportunity, working with innovation in a high-impact business, visit the <a href="https://carreiras.b2w.io/">B2W Careers portal</a>! In it, you can access all available positions. Come join our team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27403463d97c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/lambdas-with-x-ray-27403463d97c">Lambdas with X-Ray</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sharing AES Key using RSA with OpenSSL]]></title>
            <link>https://medium.com/b2w-engineering-en/sharing-aes-key-using-rsa-with-openssl-bc470afd2fb7?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/bc470afd2fb7</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[openssl]]></category>
            <category><![CDATA[cryptography]]></category>
            <category><![CDATA[rsa]]></category>
            <dc:creator><![CDATA[Guizo]]></dc:creator>
            <pubDate>Wed, 23 Sep 2020 13:05:36 GMT</pubDate>
            <atom:updated>2020-10-23T20:02:12.120Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0-kM8tn56Gv1EmmB1pl4g.png" /></figure><p>If you work with modern backend development it is inevitable that one day you’ll integrate your application with external services. When the exchanged data includes sensible information (ex: users personal information, company internal report) it is important to encrypt it to provide an additional layer of protection in the event of a leak or cyber attack.</p><p><strong>Symmetric cryptography</strong></p><p>For many years, cryptography was achieved using a shared key between two parties (sender and receiver). The key is used to encrypt a message by one party and to decrypt it by the other, hence the name symmetric. Symmetric cryptography remains widely used nowadays but it has some downsides.</p><p><strong>Symmetric cryptography downside</strong></p><p>To establish an encrypted communication using symmetric cryptography one party must generate the key and send to the other. In earlier implementations, this was achieved on a personal meeting. In the world of online services, this is not viable option as the physical distance between each party may be big and the implementation must be quick.</p><p>Sharing the key online introduces some security risks as if discovered by someone capable of intercepting the messages (ex: staff with network access), this person can view the original content, change it and even pass as the original sender without any of the parties realizing it (<a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">man-in-the-middle attack</a>).</p><p><strong>Asymmetric cryptography</strong></p><p>In asymmetric cryptography you have a public / private key pair where one is used to encrypt a message and the other to decrypt it. Currently, the most popular asymmetric cryptography algorithm is RSA (<a href="https://en.wikipedia.org/wiki/RSA_(cryptosystem)">Rivest-Shamir-Adleman</a>) which includes the following principles:</p><ol><li>The private key is private;</li><li>The public key can be shared;</li><li>The public key derives from the private key;</li><li>You cannot guess the private key from the public key;</li><li>A message encrypted with the public key can only be decrypted with the corresponding private key;</li><li>A message encrypted with the private key can only be decrypted with the corresponding public key.</li></ol><p><strong>RSA implementation</strong></p><p>Let’s say Alice wants to send a encrypted message to Bob 😅. Using RSA cryptography, each party generates a key pair and share their public key with the other. Alice encrypts her message with Bob’s public key and then with her private key before finally sending it to Bob. Bob receives the encrypted message, decrypts it with Alice’s public key and then with his private key to get to the original message.</p><p><strong>Why encrypt with private key?</strong></p><p>A very common doubt about this process is to understand why Alice would want to encrypt the message with her private key as anyone with her public key can decrypt it. The fact that the message can be decrypted with Alice’s public key assures that it was encrypted with her private key (RSA principle #6). This way, assuming that only Alice knows her private key, Bob can verify that the message was sent by Alice. This process is known as signing the message.</p><p><strong>Why not just use RSA encryption</strong></p><p>RSA was not designed to process large amounts of data. In this scenario, a symmetric algorithm like AES (<a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard">Advanced Encryption Standard</a>) is recommended.</p><p>Here is a comparison of RSA and AES algorithms used to encrypt and decrypt a 100.000 lines <em>csv</em> file with name, email and age columns using<a href="https://golang.org/"> Go</a>. Code available in<a href="https://github.com/guizoxxv/crypto-go"> this repository</a>.</p><p><strong>RSA<br></strong>Encrypt: 30.31 s<br>Decrypt: 569.62 s</p><p><strong>AES<br></strong>Encrypt: 1.36 s<br>Decrypt: 1.20 s</p><p>Cryptography was applied to the text values, not the files directly. Bear in mind that those values may vary depending on the implementation you use. In this case, it was used RSA with OAEP (<a href="https://en.wikipedia.org/wiki/Optimal_asymmetric_encryption_padding">Optimal asymmetric encryption padding</a>) and <a href="https://en.wikipedia.org/wiki/SHA-2">SHA256</a> hashing, and AES with GCM (<a href="https://en.wikipedia.org/wiki/Galois/Counter_Mode">Galois/Counter Mode</a>).</p><p><strong>AES + RSA</strong></p><p>A common way to solve this issue is to encrypt / decrypt the desired content with AES and share the AES key using RSA. This way you can take advantage of AES efficiency while minimizing its downside (sending the key).</p><p><strong>AES + RSA implementation with OpenSSL</strong></p><p><a href="https://www.openssl.org/">OpenSSL</a> is a robust, commercial-grade, full-featured toolkit for TLS (Transport Layer Security) and SSL (Secure Sockets Layer) protocols and also a general-purpose cryptography library. OpenSSL was chosen to be used in this article as it is a widely used tool, available in all platforms for many years.</p><p>Bellow are the steps for a implementation of AES and RSA to establish a encrypted communication between Alice and Bob using OpenSSL in UNIX systems.</p><p><strong>1. Generate RSA key pairs</strong></p><p>Alice generates a private key file <strong>private.pem</strong>.</p><pre>openssl genrsa -out private.pem</pre><blockquote>The recommended size for RSA keys today, considering computational power for brute force attacks, is 2048 bits, which is the default in this command.</blockquote><p>Alice extract the public key from the private key (RSA principle #3) to another file <strong>public.pem</strong>.</p><pre>openssl rsa -in private.pem -pubout -out public.pem</pre><p><strong>2. Exchange public keys</strong></p><p>Alice and Bob share their public <em>pem </em>file with each other.</p><blockquote>A 2048 bit RSA key may be reused for a long time but it is recommended to be replaced at least once a year depending on the interest of third parties to crack it.</blockquote><p><strong>3. Generate AES key</strong></p><p>In this example, the AES algorithm is used with 256 bits CBC (<a href="https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_block_chaining_%28CBC%29">Cipher-block chaining</a>) mode of operation. This mode requires a 32 bytes key and a 16 bytes initiation vector (<em>iv</em>). Alice generates a key and <em>iv</em> in hex format and store in variables for subsequent use.</p><pre>AES_KEY=$(openssl rand -hex 32)<br>AES_IV=$(openssl rand -hex 16)</pre><blockquote>Ideally you should generate a new key / <em>iv</em> pair on each use, but you can reuse the key for some time and change only the <em>iv </em>which<em> </em>does not need to be encrypted.</blockquote><blockquote><a href="https://www.openssl.org/docs/man1.1.1/man1/enc.html">OpenSSL encryption function</a> does not support GCM mode of operation.</blockquote><p><strong>4. Encrypt data with AES key</strong></p><p>Alice encrypts a text file <strong>data.csv</strong> that she wants to send to Bob.</p><pre>openssl enc -aes-256-cbc -K $AES_KEY -iv $AES_IV -in data.csv -out data.csv.enc</pre><p><strong>5. Encrypt AES key with RSA</strong></p><p>Alice encrypts the AES key using Bob’s public key to a file <strong>aes_key.enc</strong>.</p><pre>echo $AES_KEY | openssl rsautl -encrypt -pubin -inkey public.pem -oaep -out aes_key.enc</pre><p>The argument <em>-oaep </em>specifies the use of OAEP padding which is recommended.</p><p><strong>6. Sign message</strong></p><p>Alice creates a signature file <strong>aes_key.enc.sig</strong> from <strong>aes_key.enc</strong> using her private key.</p><pre>openssl dgst -sha256 -sign private.pem -out aes_key.enc.sig aes_key.enc</pre><p><strong>7. Send encrypted data, encrypted AES key, signature and <em>iv</em></strong></p><p>Alice sends the AES encrypted data file <strong>data.csv.env</strong>, the AES encrypted key <strong>aes_key.enc</strong>, the signature file <strong>aes_key.enc.sig</strong> and the plain text <em>iv </em>to Bob.</p><p><strong>8. Verify signature</strong></p><p>Bob verifies if the received data came from Alice by checking if <strong>aes_key.enc </strong>file hash matches the signature file decrypted with Alice’s public key.</p><pre>openssl dgst -sha256 -verify public.pem -signature aes_key.enc.sig aes_key.enc</pre><p>The argument <em>-sha256 </em>specify the hashing algorithm used to encrypt the file.</p><p><strong>9. Decrypt AES key</strong></p><p>Bob decrypts <strong>aes_key.enc</strong> file using his private key to get the AES key generated by Alice.</p><pre>AES_KEY=$(openssl rsautl -decrypt -inkey private.pem -in aes_key.enc)</pre><p><strong>10. Decrypt data</strong></p><p>Bob decrypts <strong>data.csv.enc</strong> file and get the original file content.</p><pre>openssl enc -d -aes-256-cbc -K $AES_KEY -iv $AES_IV -in data.csv.enc -out data.csv</pre><p><strong>Conclusion</strong></p><p>AES is a efficient algorithm for large files cryptography but sharing the key is a security risk. One way to minimize this risk is to use RSA algorithm to send the key. OpenSSL provides a easy cryptography implementation for this logic that can be automated.</p><p>If you are looking for a development opportunity, working with innovation in a high impact business, access the <a href="https://carreiras.b2w.io/">B2W Carreiras</a> portal! Access and find out all available positions. Come join our team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc470afd2fb7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/sharing-aes-key-using-rsa-with-openssl-bc470afd2fb7">Sharing AES Key using RSA with OpenSSL</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Technical SEO: an experiment to optimize Crawl Budget in big ecommerce sites]]></title>
            <link>https://medium.com/b2w-engineering-en/technical-seo-an-experiment-to-optimize-crawl-budget-in-big-ecommerce-sites-640c11b46e7b?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/640c11b46e7b</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[seo]]></category>
            <category><![CDATA[crawl-budget]]></category>
            <category><![CDATA[google-crawl-budget]]></category>
            <category><![CDATA[robotstxt]]></category>
            <dc:creator><![CDATA[Adriano Rodrigues]]></dc:creator>
            <pubDate>Tue, 18 Aug 2020 12:40:51 GMT</pubDate>
            <atom:updated>2021-06-18T13:38:12.886Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3sSRa7qWShDn1uA9iXWjtQ.png" /></figure><p>Every day, search engines robots’ crawl the web searching for new websites, they discover new content, while collecting all this information into a huge index or “library”.</p><p>To access all the content on the web, these robots “click” on every link they find, and in this way, they move around the web.</p><p>The search for the best and most up-to-date content is a continuous job that requires a lot of resources.</p><p>New information appears all the time on the internet and robots need to revisit sites frequently, in order to ensure that content is always up to date. This way, it will reflect the best response to the user’s search.</p><p>To understand how search engines decide on how and when these robots visit your site, it is necessary to talk about “Crawl Budget”.</p><h3>1. What is Crawl Budget?</h3><p>Crawl Budget is the number of requests a search engine makes to one domain in a set period. Most sites don’t need to worry about Crawl Budget, once it will not be an issue unless the website has an enormous number of pages.</p><p>As <a href="https://twitter.com/methode">Gary Illyes</a> explained in the <a href="https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html">Google Webmaster Central Blog</a>:</p><blockquote>“If a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently.”</blockquote><p>In websites with millions of URLs, crawling can be a challenge. In this case, control of when and how many accesses are allowed to the robots is very relevant, as excessive access can affect server performance or hinder user experience.</p><blockquote>Gary Illyes, in his article for the Official Google blog, states that the Crawl Budget is defined by two concepts: Craw rate limit and Crawl demand.</blockquote><p><strong>Crawl rate limit </strong>refers to the number of parallel connections that robots make in order to crawl your site and the waiting time between calls. This limit may vary depending on the site&#39;s health or performance. For example, if a site responds quickly, this limit may increase automatically.</p><p>However, if a site is slow or has errors, the crawl rate may be limited. One way to modify the crawl rate limit is by configuring the crawl rate in Google Search Console.</p><p>However, increasing the crawl rate limit in Google Search Console does not guarantee that the Crawl Budget will increase.</p><p><strong>Crawl demand </strong>is a measure of the importance of your site to Google and comprises two factors: popularity and recency. A popular URL tends to be crawled more often, to ensure that the content is up to date. Changes to site content may also cause an increase in crawl demand.</p><p>Google’s definition:</p><blockquote>“Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.”</blockquote><h3><strong>2. Crawl Budget Optimization</strong></h3><p>Several factors can negatively affect the robot’s behavior, including loading speed, 4XX access errors, duplicate content, excessive usage of parameters and filters.</p><p>Adding filters or parameters on very large websites, there is a possibility of having duplicate content, such as in product pages, for example.</p><p>With ecommerce websites, product offerings can run into the millions of items, generating an almost infinite number of URLs due to the various combinations of filters and parameters.</p><p>Also, it is important to understand which pages and how many times the robot is visiting.</p><p>For example, if a website has 1,000 pages and the robot finds 1,000 URLs every day, it could be that the crawl budget is perfectly optimized, or it could mean that Google is hitting the same page 1,000 times.</p><p>Another important factor is the server capacity. There is no point in receiving thousands of visits from the robot each day if this hinders the site’s UX.</p><p>One of the ways to control and optimize the Crawl Budget is to create a txt file with guidelines for crawling the website. Robots.txt files are created at the root domain level, for example, <a href="http://www.yourdomain.com.br/robots.txt">www.yourdomain.com.br/robots.txt</a>.</p><p>The robots.txt file defines the instructions that robots follow when navigating the website, such as, which pages they should access. A well-structured robots.txt file is crucial for helping robots find the right content.</p><p>Example of a snippet from Google’s robots.txt, that can be found at <a href="https://www.google.com/robots.txt">https://www.google.com/robots.txt</a>:</p><pre>User-agent: *<br>Disallow: /search<br>Allow: /search/about<br>Allow: /search/static<br>Allow: /search/howsearchworks<br>Disallow: /sdch<br>Disallow: /groups<br>Disallow: /index.html?<br>Disallow: /?<br>Allow: /?hl=<br>Disallow: /?hl=*&amp;<br>Allow: /?hl=*&amp;gws_rd=ssl$<br>Disallow: /?hl=*&amp;*&amp;gws_rd=ssl</pre><h3>3. Case: Black Friday in B2W</h3><p>As stated, “demand” is a relevant aspect of Crawl Budget. If Google understands a website as important for users, it will increase the frequency of robots visits.</p><p>On important dates such as Black Friday, search engines aim to provide all the best and latest content on the web and to achieve this the crawl demand increases.</p><p>Black Friday is one of the most important shopping events in the world, featuring some of the biggest promotions of the shopping year. Many customers wait for this date to buy higher value items or to start Christmas shopping.</p><p>During this period, B2W’s e-commerce websites (<a href="https://www.americanas.com.br/">Americanas.com</a>, <a href="https://www.shoptime.com.br/">Shoptime</a>, <a href="https://www.submarino.com.br/">Submarino</a> and <a href="https://www.soubarato.com.br/">SouBarato</a>) have an increase inthe number of visitors and, simultaneously, web crawlers also want to access the pages more frequently in order to keep content updated.</p><p>Analyzing the log files from Black Friday 2018, it was noticed a peak in the number of accesses from <a href="https://support.google.com/webmasters/answer/182072?hl=en">Googlebot</a>, Google’s robot that collects site information.</p><p>For 2019, the goal was to reduce the volume of crawler requests in the days preceding and over the Black Friday weekend. This experiment aimed to preserve the website performance and provide optimal shopping experience for the visitors.</p><h4><strong>3.1. The experiment</strong></h4><p>Before Black Friday it was applied a restriction in the robots.txt file so that certain pages could not be found by search engines. However, the policy was not too restrictive and it still allowed a large number of pages with filters to be found.</p><blockquote><em>In 2019 there was an increase in bots access before Black Friday, starting on November 23rd. Between November 22nd and 25th, there was an 840% increase in the number of Googlebot accesses.</em></blockquote><p>To reduce the growing number of non-human accesses and to preserve the user experience on the websites, it was planned to restrict crawler access via robots.txt.</p><p>In parallel, the crawl rate was reduced in Google Search Console. If the bot access was successfully contained, the robots.txt rules should be modified again, in order to gradually give the crawlers more freedom to navigate.</p><p>In the first stage of the experiment, only the top-level pages of each domain were allowed and access to any pages with a filter was blocked.</p><blockquote><em>The rule was applied on Wednesday, November 27th, at 18:50 (UTC-3) and within 30 minutes we saw a drastic reduction in the number of crawler accesses.</em></blockquote><p><em>About an hour after the change, the total volume of hourly accesses was reduced by 95%, dropping bots accesses from 400k req/hour to around 20k req/hour. The access amounts continued to reduce and on Thursday, 28th, it returned to a normal level as experienced before the peak.</em></p><p>The image below shows the number of accesses of an API during the experiment period. After the new robots.txt policy was implemented, the large reduction in the number of robots’ accesses was seen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*tAz0CAIyk04hp9yOcP5hbA.jpeg" /><figcaption>Crawlers’ accesses to americanas.com.br a day before Black Friday</figcaption></figure><p>B2W’s tech ecosystem has around one thousand APIs, therefore controlling output is crucial to preserve user experience during high traffic moments. The goal was to reduce the output and to maintain on-prem infrastructure to process crawler requests.</p><p>The image below shows the throughput of an API that it’s directly connected to crawlers’ accesses during the experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*uRB7jLxUgfDUn_9ZUpqY3w.png" /><figcaption>Throughput of an API directly connected to crawlers’ accesses</figcaption></figure><p>After Black Friday, on December 2nd, at 15:30, it was applied a new rule in the robots.txt file to allow crawlers to navigate all domains, including specific combinations of filters.</p><p>In the days that followed, it was experienced an increase in the number of requests, but it was near to the normal level.</p><h4><strong>3.2. Possible side effects</strong></h4><p>An important point was to assess and understand other aspects and KPIs that may have suffered side effects.</p><p>To understand other variables when changing the robots.txt file, the following metrics were evaluated: number of indexed pages, weighted average position, number of pages and number of keywords in the top 10 positions.</p><p>None of these metrics showed significant variation or indicated any turbulence during the period of the experiment, validating the effectiveness of the change in the Robots.txt file.</p><h3><strong>Final considerations</strong></h3><p>For any e-commerce website, it is essential to experiment in order to evolve and guarantee the best user experience and business growth.</p><p>On large sites, with hundreds of millions of pages, it is necessary to closely watch the volume of accesses received, to ensure the smooth functioning of the servers.</p><p>This experiment brought positive results and it was possible to limit robot access, therefore, ensuring the best user experience during the Black Friday event!</p><p><a href="https://medium.com/@adriano_acr/tech-seo-experimento-para-otimiza%C3%A7%C3%A3o-de-crawl-budget-em-grandes-ecomerces-bd03653d3d26">Read the text in the Portuguese version</a>.</p><p><em>I hope you enjoyed this article and that it can help you and your business. I would love to hear your feedback, feel free to send me a message through </em><a href="https://br.linkedin.com/in/adrianocrodrigues"><em>Linkedin</em></a><em>!</em></p><p><em>I would like to thank </em><a href="https://medium.com/@RFenning"><em>Richard Fenning</em></a><em> for the co-authorship of the translation of this article; to</em><a href="https://medium.com/u/99a2f48cdfa0?source=post_page-----fd392ead267f----------------------"><em> </em></a><a href="https://medium.com/@pedrogil.alcantara"><em>Pedro Gil Alcantara</em></a><em> for his partnership for the creation of the above Robots experiment and to my colleagues from the SEO Team here in B2W for reading and reviewing this article, and especially to </em><a href="https://medium.com/@tfandrade"><em>Tiago Andrade</em></a><em> for his support and suggestions. Thank you!!</em></p><p><em>Anyone who is interested in this topic or wants to continue the discussion or simply to contribute to the community, we have two LinkedIn groups:</em><a href="https://www.linkedin.com/groups/8792524/"><em> SEO in RJ</em></a><em> (Rio de Janeiro) an</em><a href="https://www.linkedin.com/groups/8794711/"><em> SEO in SP</em></a><em> (São Paulo).</em></p><p>If you are looking for a development opportunity, working with innovation in a high-impact business, visit the <a href="https://carreiras.b2w.io/">B2W Careers portal</a>! In it, you can access all available positions. Come join our team!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=640c11b46e7b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/technical-seo-an-experiment-to-optimize-crawl-budget-in-big-ecommerce-sites-640c11b46e7b">Technical SEO: an experiment to optimize Crawl Budget in big ecommerce sites</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[restQL: Tackling microservice query complexity]]></title>
            <link>https://medium.com/b2w-engineering-en/restql-tackling-microservice-query-complexity-27def5d09b40?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/27def5d09b40</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[americanas tech]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 20:50:33 GMT</pubDate>
            <atom:updated>2020-10-23T20:19:46.414Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://medium.com/u/6b521c5eb9b1">Ricardo Mayerhofer</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zUs3KLWZxMlZzXDHDKSyDA.png" /></figure><p>Four years ago B2W, the largest e-commerce operation in Latin America, started a microservice migration. It was a project that took 3 months to generate its first results and 3 years to be completed.</p><p>From a technical perspective, the benefits to the teams responsible for the microservices were clear from the first deploys. With the new architecture, even though they had more applications to handle, they were smaller which meant: easier maintenance, technology freedom and deploy risk mitigation.</p><p>Nonetheless, for the microservices consumers the architecture went not so smoothly. The front layer developers (website and apps) started to raise issues in connection with application performance and code complexity.</p><p>Previously they had to deal with few services to get the required information to render a screen. A common approach was to have a back-end service that would gather all or most of the information needed. In this new scenario the information was spread throughout different microservices and had to be orchestrated.</p><p>To give an example, let’s consider a payment screen from an e-commerce website:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*aZ_mxj_JyHKyZTR0GPSg7A.png" /></figure><p>On a monolithic website, a common approach is to make a route available which returns the whole page HTML or all the data required for the initial render.</p><p>However, in a microservice architecture there are multiple services, each one of them handling a single concern. In the above example, there’s one service for the basket, another for customer address, another for shipping and payment options. This architecture ends up generating the following request scenarios.</p><h4>Multiple parallel calls</h4><p>A complex application using a microservice architecture may have dozens of parallel requests to get the render information. Add to the previous example: gift packaging, recommendation, upsell, etc. This is a problem for application running under HTTP 1.X where browsers limit the number of parallel requests made to the same domain, thus impacting the application performance.</p><h4>Chained calls (1+n request problem)</h4><p>Often a microservice requires data from another microservice to serve its request. For example, to calculate the shipping fee the service needs the products, returned from the cart service, and the zip code, returned from the address service.</p><p>If one considers the average latency to a datacenter being 100ms, in a chained invocation of 3 calls a 300ms performance penalty is being paid, only because of network round-trips. This scenario is made worse in mobile networks where there’s a big variance in the network latency.</p><p>From a development perspective, the chained invocations generate code complexity. For example, in a Javascript application promises of promises start to pop up.</p><h3>Existing solution</h3><p>We quickly moved to find possible solutions. Our best options seemed to be <a href="https://github.com/Netflix/falcor">Falcor</a> and <a href="http://graphql.org/learn/">GraphQL</a>.</p><p>At that time Falcor, from Netflix, already had been announced but not yet open sourced. Its goal was to make it easier to fetch information scattered in different places, thereby solving the code complexity problem. Falcor was also able to optimize the requests so some extend (deduping and batching it), lessening the round-trip problem.</p><p>However it was (and still is) only available to Javascript clients, which eliminated the use in our mobile apps written in Objective-C (iOS) and Java (Android).</p><p>GraphQL offered a mature language to query data from services and a growing ecosystem.</p><p>However, unlike REST, GraphQL didn’t leverage the many HTTP based features developed across the years. Two things we considered important:</p><h4><strong>Caching</strong></h4><p>GraphQL was initially developed by Facebook where every request/response is highly customized. Every user sees a different mix of information, which means equal requests/responses are seldom.</p><p>Having said that, caching, specially network caching (CDN, proxy cache like Varnish, Squid) is not crucial.</p><p>On the other hand, a public website, like an e-commerce, will have many users visiting the same information (e.g. a product page). This makes caching, specially network caching, very efficient.</p><p>Because GraphQL will usually work over POST method and always returns HTTP success (200), it’s not possible to take advantage of network caching and native browser caching won’t work either.</p><h4><strong>HTTP return codes</strong></h4><p>HTTP status code provides a very standardized way to communicate different errors and success scenarios, which makes it easier for the client to figure out what’s happening in a human and machine readable way.</p><p>This allows the developer to navigate between different APIs and contents, only having to learn about the content itself.</p><p>The machine readable nature enables automatic handling of different scenarios, which in turn improves resilience. For instance it’s common for a HTTP client to retry a request upon receiving a 5XX response, but it will give up after a 4XX response.</p><h3>restQL</h3><p>In view of our situation, in an internal hackathon was developed what came to be called restQL.<a href="http://restql.b2w.io/"> restQL</a> is a query language for microservices that makes it easier to fetch information from multiple services in the most efficient manner.</p><p>It aims to integrate seamlessly with a REST microservice architecture. From the client perspective, restQL Server is like any other REST service, so it doesn’t require any special access or specific client.</p><p>restQL now is an open-source project available under MIT license.</p><h4>Query Language</h4><p>A restQL query is expressed using a statement describing the resources to be fetched and its parameters:</p><pre>from product</pre><pre>    with</pre><pre>        productId = &quot;156AB&quot;</pre><p>This query format allows easy use of common service orchestration scenarios.</p><h4>Chained and parallel calls</h4><p>Upon receiving a query, restQL builds a dependency graph of the listed resources. Those resources which have no dependency are executed in parallel and those which have dependency are parked until the dependency returns.</p><p>In the example below we have a query free from dependencies. In in this case both are going to be executed in parallel.</p><pre>from productSearch</pre><pre>    with</pre><pre>        productName = “TV”</pre><pre>from customer</pre><pre>    with</pre><pre>        customerName = “Joe”</pre><p>In the query below we have an example of a chained request. In this case restQL will wait for the customer service response and right after it call the customerPurchase service passing on the customer information as a parameter.</p><pre>from customer</pre><pre>    with</pre><pre>        customerName = “Joe”</pre><pre>from customerPurchases</pre><pre>    with</pre><pre>        customerId = customer.id</pre><h4>Multiplexed call</h4><p>A common scenario when orchestrating microservices is when a service returns a list and for each item of that list another service invocation is required. For instance, the product search service may return the product ids and it’s necessary to call another service to get the details of each product (price tag, images, etc.).</p><p>restQL syntax is the same for simple and multiplexed call. If the return of the first call is a simple value, restQL will perform one call. If it’s a list, it will multiplex the call, making one call for each value on the list.</p><pre>from productSearch</pre><pre>    with</pre><pre>        productName = “TV”</pre><pre>from product</pre><pre>    with</pre><pre>        productId = productSearch.result.productId</pre><h4>Filtering</h4><p>We can use restQL query language to filter the resource response, thus reducing payload for the client. This is useful when the API doesn’t implement filtering itself. Filtering is achieved by using <em>only</em> modifier.</p><pre>from product</pre><pre>    with</pre><pre>        productId = “156AB”</pre><pre>    only</pre><pre>        title</pre><h4>The server</h4><p>restQL server is the application which handles restQL queries. restQL server is agnostic of the underlying services, which means it doesn’t hold any service schema or special ways of invocation. It acts like a bridge between the client and the backend APIs.</p><p>The only information stored in restQL server is the resource name and its invocation end-point.</p><p>restQL server is implemented in Clojure using HTTP Kit and Clojure CSP implementation (core.async).</p><p>For clients in the JVM world it’s possible to use directly restQL core, writing and executing restQL queries directly without the need of an additional server.</p><h3>REST impedance mismatch</h3><p>There’s an impedance mismatch in a solution like restQL, where one service invokes another N services. To keep compatibility with a REST architecture some topics need to be addressed.</p><h4>HTTP Status Code</h4><p>Because restQL may invoke many services in the same query, what should be done if one service returns success code (200) and another one in the same query returns error code (500)?</p><p>Given the practical implications, if restQL returned 200 in this case, the intermediate proxies would consider a success HTTP call and would potentially cache the request, carrying the error to the whole user base. For HTTP clients, automatic retry wouldn’t work.</p><p>To address this issue, restQL always returns the highest HTTP code from the invoked services as the query response status code.</p><p>In the same query there may be essential and non-essential resources. Think about a query fetching a cart and its related recommendations. It’s reasonable to show the cart even though we were not able to fetch the recommendations. In such cases it’s possible to use the<em> ignore-errors</em> modifier in the non-essential resources. When this happens, even if the call to this specific resource fails, restQL will return 200. Even in these circumstances the client can still manually check the restQL response to find out if the request was successful. This status code information is included as a metadata in the response and can also be used for debugging and troubleshooting purposes.</p><h4>Cache Control</h4><p>Cache-control is a header returned in a HTTP call that tells the intermediate proxies and the end client on how to handle the cache of the returned content . For example: a cache-control header with max-age=60 tells that the client can safely cache the request for 1 minute.</p><p>To avoid stale data, restQL considers the smaller max-age among the services responses to use in the query response.</p><p>It’s also possible to configure the cache-control for a specific query using the directive <em>use </em>inside the query.</p><h4>Timeout</h4><p>Each microservice in a query may have a different SLA, which means different response times.</p><p>When each microservice is called isolatedly, it’s easy to configure the timeout of each call as most HTTP clients allow the configuration of such setting.</p><p>In case of an orchestrator like restQL it’s necessary to implement some way to fine-grain control the timeout of the underlying invocation. In restQL this is achieved by the <em>timeout </em>modifier in the query, which specifies the timeout of each resource. If timeout triggers, a 408 (timeout) status code is assigned to the resource.</p><h4>Query size</h4><p>Although URI RFC doesn’t set any limits to the URL length, in practice some browsers and proxies (e.g. Apache, Nginx) won’t work with long URLs. For example, Nginx default URL length limit is 8k, after which it returns 413 (entity too large). This may be a problem if the query is too long.</p><p>To work around this issue restQL has saved queries. Saved queries is a way for clients to previously save a query in the server and invoke it by just referencing its name and revision.</p><p>Each saved query update will generate a new query revision and keep the existing one untouched. This avoid consumers being broken down and makes the cache of the query statement retrieval on the server very efficient (you can actually cache it forever).</p><h4>Conclusion</h4><p>As we’ve seen, a microservice-based architecture isn’t a silver bullet. Despite its immediate benefits to the service teams, it poses by its very nature a challenge in terms of complexity and performance to the consumers.</p><p>Fortunately there are some technologies available to overcome those problems, worth mentioning Falcor, GraphQL and the recently introduced restQL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27def5d09b40" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/restql-tackling-microservice-query-complexity-27def5d09b40">restQL: Tackling microservice query complexity</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using service graphs to reduce MTTR in a HTTP-based architecture]]></title>
            <link>https://medium.com/b2w-engineering-en/using-service-graphs-to-reduce-mttr-in-a-http-based-architecture-bb77decdd9a1?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb77decdd9a1</guid>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[americanas tech]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 19:13:54 GMT</pubDate>
            <atom:updated>2021-06-18T13:37:11.545Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://medium.com/@marcus.cavalcanti">Marcus Cavalcanti</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtU-pVliFkQTPkkBbeBmKQ.png" /></figure><p>(source: leica-geosystems.com)</p><p>Microservices is a new trend in software development which many organizations around the world are adopting (and counting…). From e-commerce websites, to social network platforms, everybody is talking about this new way to design software architectures. If you have premises like scalability and high availability or if you want to <a href="https://www.martinfowler.com/articles/microservices.html#OrganizedAroundBusinessCapabilities">reorganize your teams around business capabilities</a>, this new approach can help you. A lot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JgAqRgmnPNFwX1pT.png" /></figure><p>The growth of the “Microservices” term in Google Search in the last 5 years (source: Google Trends)</p><p>At B2W, we adopted this new paradigm in mid/2014 and since that, we have had enormous gains on our projects, but with a price. Sometimes with a huge price. Microservices brings new challenges, and in my opinion, dealing with many dependencies is the biggest one. As your business grows, you’ll probably have more new services over time. Consequently, problems related to monitoring and operating the platform will also grow exponentially at the same rate.</p><p>If you didn’t architect your platform considering basic aspects such as <a href="https://principlesofchaos.org/">resiliency</a>, it’s not rare that one component (Microservice) breaks the whole system and the consequence can be thousands, or even millions of dollars in losses.</p><p>Monitoring is always a good practice. You have to do it. But how can you monitor a hundred of Microservices in an effective manner? How do you know if a service in the end of your dependency chain is impacting some other services in the beginning? If you don’t have these answers fast, <strong>more time you’ll demand to react</strong>.</p><p>A situation that we must avoid at B2W is stop selling. When something goes wrong we need to know what happened as soon as possible. But as we have a lot of Microservices, sometimes find the problem is like to find a needle in a haystack… a real nightmare!</p><p>We have been using a lot of monitoring and alert tools to help us in the day-to-day operation. We also use logging extensively, recording every relevant information in our <a href="https://www.elastic.co/elk-stack">ELK stack</a>. But as I said before, in a Microservice architecture the whole is important. Even if you designed your <a href="https://medium.com/b2w-engineering/lessons-learned-about-run-microservices-b360347c8a77">architecture to be resilient</a>, there are some moments where a unified vision of the whole is the most important. Even more when you have a symptom like an alarm in many applications/services at the same time.</p><p>Last year, I watched a presentation of a Brazilian guy (<a href="https://www.linkedin.com/in/eduardosaito">Eduardo Saito</a>) that works for LinkedIn. In his presentation, he talked about the challenges of operating the LinkedIn platform. They have a complex process to avoid risks when deploying service updates in production, and to help them they developed an internal tool called inVisualize.</p><p>InVisualize is a visual tool intended to provide insights over LinkedIn’s platform health and, inspired by that, I started to think about how we can implement this at B2W.</p><p>B2W is an e-commerce platform and our revenue depends a lot on our availability. We must be online 24x7. But reality is tough and as <a href="https://www.allthingsdistributed.com/">Werner Vogels</a> says:</p><blockquote><a href="https://www.slideshare.net/AmazonWebServices/high-availability-websites-part-one/12-Everything_fails_all_the_time"><em>“Everything fails all the time”</em></a></blockquote><p>There is a specific measure that is very common when we’re talking about maintainability, it is called <a href="https://en.wikipedia.org/wiki/Mean_time_to_repair">MTTR</a> (mean time to repair).</p><p>It means how long does it take to solve a problem, or how long we take to identify an unexpected situation and fixed it.</p><p>Setted by this mindset provided by Mr. Vogels and considering that our MTTR must be as lower as possible, I get back to the question: how can we monitor a hundred of Microservices in an efficient manner?</p><p>I’m pretty sure that visual representations are much more human-readable than textual representations, so my idea was create a visual representation of our architecture considering key aspects that we use to infer a problem, or better, a unhealthy situation.</p><p>When our platform is degrading, usually we consider three aspects:</p><ul><li><strong>Latency:</strong> response time between two services;</li><li><strong>Error rate:</strong> HTTP 5** family errors;</li><li><strong>Throughput:</strong> the number of requests from one service to another in a given period.</li></ul><p>The throughput metric alone doesn’t mean so much isolatedly, but it can be the cause of a high latency scenario. In the same way, a high latency scenario can explain a high error rate scenario.</p><p>But how do we know it? Using levels. For each indicator is possible to have three different levels (one at a time): <strong>normal</strong>, <strong>warning</strong> and <strong>critical</strong>. They are self-explanatory.</p><p>To represent a relationship between two services, we use graphs. A <a href="https://en.wikipedia.org/wiki/Graph_%28abstract_data_type%29">Grap</a>h, is a very appropriated data type to represent relationships, and in our case, what we want is to represent the relationship among services (N:N).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*tfnd00Yal17_rnRn.png" /></figure><p>A simple graph relationship (source: <a href="http://wikipedia.com/">Wikipedia</a>)</p><p>And what about the visual representation of each indicator mixing with levels? That’s a good question.</p><p>For <strong>latency</strong>, we’re using the line type. If the line is not dotted the situation is considered normal. If the line is dotted, but with small spaces, we consider a warning situation. And if the line is dotted with long spaces, probably we’re in a critical situation. For <strong>error rate</strong>, we’re using the line color. Blue represents normal, yellow represents warning and red represents a critical situation. And finally, for the <strong>throughput</strong> indicator, we’re using the line fill. The thicker the line, more latency (critical) we have.</p><p>InnerSection is our answer to identify suspicious/strange behaviours in B2W’s Microservices ecosystem, reducing MTTR.</p><p>We must avoid scenarios like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/0*bMGmchz7MTR2S1PE.png" /></figure><p>When an e-commerce portal stop selling, that’s the sensation (source: ripe)</p><p>During one of our <a href="http://ironbit.b2w.io/">Hackathons</a>, I joined with 4 brilliant guys in a team to develop InnerSection in 24 hours. The outcomes were really incredible.</p><p>Our mission were to create a holistic vision represented by a graph of our platform, containing all Microservices running in production. To confirm my theory I started to think about all the limitations that we had without InnerSection, and that’s what I got:</p><ul><li>Alarms work very well, but they need to exists;</li><li>APM’s are also good, but we must know where is the problem;</li><li>Logs are necessary, however all applications must log in a specific pattern, otherwise, will not be useful to correlate problems;</li><li>As more service we have, more difficult will be to detect problems, then a visual representation looks interesting.</li></ul><p>With InnerSection we addressed all limitations above, then we consider simple decisions:</p><ul><li>Visual representation must be a <strong>services graph;</strong></li><li>Three <strong>indicators</strong> that we use to infer a problem: error rate, latency, throughput;</li><li>Three <strong>levels</strong> for each indicator: normal, warning and critical;</li><li>Well defined <strong>symbologies</strong> per <strong>indicator</strong> and <strong>levels.</strong></li><li>Isolation: agent must be isolated at application/service level;</li><li>Easy to plug: minimum of configuration;</li><li>Agnostic: it does not depend on a specific technology or infrastructure design;</li><li>Not intrusive: can not impact the application;</li><li>Without customization: you don’t need to do anything in the application.</li></ul><p>The main flow of InnerSection is about listen all TCP packets in a given machine, identify which are HTTP and then send them to an API.</p><p>This approach allow us, to still use the best technology to solve a specific problem. Technology agnosticism is one of our premises in B2W.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/0*LHLFSVdLmX5qzVBK.png" /></figure><p>InnerSection’s Stack — Big Picture</p><p>Below, the four main components of InnerSection’s architecture.</p><p>The main componente is the agent, that must run in the same VM of the application.</p><p>The agent, developed in GoLang, monitors all TCP Packets in a machine and identify which are HTTP. We’re doing this using <a href="http://www.tcpdump.org/">libpcap</a>, same that <a href="https://www.tcpdump.org/tcpdump_man.html">tcpdump</a> uses.</p><p>This agent has the ability to store all packets/events in a local buffer, in memory, and when a size/bytes or a time period is reached, it flush them.</p><p>A flush is an action about send a set of events in a specific pre-formatted JSON to a HTTP API, that we call Ingest API.</p><p>Using an agent helped us to keep our premises related to agnosticism, isolation, and be easy to plug and not intrusive. The agent uses API’s hostname to identify the target, and an OS environment variable to identify which API is the source.</p><p>This API receives all events sent by agents and persists in a MongoDB database. We opted for MongoDB for many reasons: the team already had a knowledge about how to operate Mongo, it is schemaless, works natively with JSON, is easy to filter and aggregate data, and also because Mongo has a built-in graph model.</p><p>We’re using the <a href="http://mongodb.github.io/mongo-java-driver/3.0/driver-async/">async java driver</a> of MongoDB. It works with callbacks to ensure that all operations in the database are non-blocking. In the same way, the Ingest API is async too, so every call returns a 202 HTTP status code, and another thread do the hard work to persist the data.</p><p>We have plans to use a message broker, like Kafka or Kinesis, to decouple and scale better the production of the consumption of data.</p><p>This API is responsible to retrieve all data persisted in MongoDB in a given period. This data is returned already preformatted in a way that SigmaJS can understand. Graph API also apply some math formulas to generate a score (level) for each indicator.</p><p>We decided to calculate each score in this API rather than Ingest API, because if we change some business rule associated to the score calculation, we wouldn’t need to reprocess the entire dataset.</p><p>Our frontend was developed using <a href="http://sigmajs.org/">SigmaJS</a>, a JS library that generates graphs from a JSON.</p><p>The frontend constantly pool new fresh data from Graph API, is like a living organism.</p><p>I know that at this point, maybe you are thinking about a better and efficient way to retrieve data. We thought the same. Something bidirectional makes sense for this use case, like Websocket.</p><p>Our idea in the long term is to use some ML model to learn with the past and predict the future. Currently the formula is very simple, but meets our needs.</p><p>There are two important things here. The first thing is that all relationships are 1:1, that is, caller (API A)-&gt; provider (API B). All indicators (error rate, latency and throughput) and levels are also defined per relationship. This is how data is persisted in MongoDB.</p><p>The second thing is that some indicators are fixed, based on our experience and observation acquired by empirical knowledge.</p><p>Considering that, we have:</p><p>normal ≤ 3%, warning &gt; normal &lt; critical, critical &gt; 10%</p><p>normal ≤ 45ms, warning &gt; normal &lt; critical, critical &gt; 150ms</p><p>Is not easy to identify in an accurate way the throughput. There are many situations that we must be aware, for example, if a new Microservice is deployed.</p><p>We consider the throughput (service calls) of all the services in the graph and from this we define three levels based on the median.</p><p>To increase the range of the “warning” group, that is the median, we applied a factor of 1.01. The same for the “critical” group, where we applied a factor of 1.05.</p><p>To clarify what I’m saying, please consider this specific set (RPM K): {190, 100, 80, 55, 45, 10, 5}.</p><p>The median of this given set should be 55, but to avoid outliers, we applied the average of all numbers after the median (critical group), in this case (190,100,80). The average should be: 90.</p><p>Considering the rule above and applying factors, the three levels will be like this: normal &lt; warning (55550 RPM) &lt; critical (94500 RPM)</p><p>This is a screenshot of InnerSection using some test data.</p><p>Each node in the graph is a different Microservice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*lFJWetotJUGJY_FG.jpeg" /></figure><p>InnerSection Working! (data is not from production environments)</p><p>Normally in a Microservice architecture, databases are a dependency of a service, so if some database is slow, this impact will probably be reflected in the response time of a Microservice.</p><p>Futhermore, would be insane to understand and monitor each protocol of any dependency (filesystem, message brokers, etc) of a Microservice.</p><p>We don’t have plans to adapt InnerSection to work with Lambda Functions, it would be hard as we don’t have direct access to the machines where Lambdas are running. But if you’re looking for a solution to work with serverless paradigm, I recommend <a href="https://aws.amazon.com/en/xray/">X-Ray from AWS</a>; in case of your Lambdas are already deployed in AWS.</p><p>As I mentioned before, we have plans to use ML models to predict the score levels of each indicator. I pretty sure that will turn the model more accurate.</p><p>Another important thing is to support containers running in a host machine. As in a container normally you only have one process, a good feature can be run the agent from the host machine.</p><p>We’re also working on have a better way to deal with applications that don’t have a semantic hostname. In this case, is difficult to identify the application in the service graph. Currently, you can configure this option through a properties file, but ideally it should be automatic.</p><p>Finally, the main idea is to open InnerSection to the community, but some fine tuning is necessary before release a public version.</p><p><em>Originally published at </em><a href="https://medium.com/@marcus.cavalcanti/using-service-graphs-to-reduce-mttr-in-a-http-based-architecture-624d2c9d54c1"><em>medium.com</em></a><em> on March 23, 2018.</em></p><h3>If you are looking for a development opportunity, working with innovation in a high-impact business, visit the <a href="https://carreiras.b2w.io/">B2W Careers portal</a>! In it, you can access all available positions. Come join our team!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb77decdd9a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/using-service-graphs-to-reduce-mttr-in-a-http-based-architecture-bb77decdd9a1">Using service graphs to reduce MTTR in a HTTP-based architecture</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Microservices with AWS Fargate]]></title>
            <link>https://medium.com/b2w-engineering-en/deploying-microservices-with-aws-fargate-3760820b5d09?source=rss----b0bc3e71822e---4</link>
            <guid isPermaLink="false">https://medium.com/p/3760820b5d09</guid>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[americanas tech]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 19:09:09 GMT</pubDate>
            <atom:updated>2020-10-23T20:38:26.116Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://medium.com/@marcus.cavalcanti">Marcus Cavalcanti</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYmRHwON7OjHVLrx8r-VzA.png" /></figure><p>Recently, in the last re:Invent (2017), AWS launched a new service called <a href="https://aws.amazon.com/blogs/aws/aws-fargate/">Fargate</a>, which can simple be described as an abstraction to deploy applications through containers.</p><p>AWS already had a service to manage containers, called <a href="https://aws.amazon.com/ecs/">Amazon Elastic Container Service</a> (ECS), but ECS is more connected with the infra-structure layer than with the application layer.</p><p>The main idea behind Fargate is that you only have to think about the computational footprint of your application, memory and CPU, and it will manage all the other stuff for you automatically.</p><p>In B2W, we have many environments where we run our Microservices. One of these environments is AWS, and we deploy our applications using Elastic Beanstalk. Another environment is managed by us, and to simplify our operation, B2W’s DevOps team built an abstraction that uses <a href="http://mesos.apache.org/#program-against-your-datacenterlike-it%E2%80%99s-a-single-pool-of-reso">Mesos</a> and other Mesos frameworks. The concept of Mesos is to provide a unified vision of your infrastructure and their compute resources, so you can organize all of those resources through Cluster(s).</p><p>For task scheduling, like deploying a new version of a Microservice, we use a Mesos framework called <a href="https://mesosphere.github.io/marathon/">Marathon</a>. The role of Marathon in this process, is to deploy the new service, check if it is healthy and register it in a Load Balancer, in this case Traefik, through a integration that uses Marathon as a <a href="https://docs.traefik.io/configuration/backends/marathon/">backend configuration engine</a>. A scheduled task for Marathon is a container running with a<em> /health</em> endpoint where Marathon can check the application’s heartbeat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*sbNmxG1UNnRSBhIJ.png" /></figure><p>Deploying and managing an application in ECS with EC2 x Fargate</p><p>But in AWS, we don’t use containers, and this leads to some problems, like:</p><ol><li>Our CD process varies. In one environment (AWS), our deploy artifact is a FAT JAR, and in another one (Mesos), our deploy artifact is a Docker container;</li><li>Our troubleshooting process is also different. In AWS, we have a virtual machine (EC2) just running a Java Application. In Mesos we have a different scenario, with many containers (applications) in the same VM. In practice, this is quite different because your application shares some OS resources with others, like file descriptors;</li><li>Deploy containers is better. The startup time is much faster and you have the guarantee that if the container is running in your machine then it will run in any environment.</li></ol><p>For the reasons above, using containers at AWS seemed like an obvious path for us. We started trying using ECS, but it is a step before Marathon; ECS is more like Mesos than Marathon. As we are responsible for applications, not for machine level resources, Marathon is closer to our reality, where each team is responsible for (Micro)services.</p><p>If you want to benefit using Fargate, all the steps involved in the process are described below. For this, I will create a Microservice, a Docker image of this service, push it to ECR, and deploy in AWS using Fargate.</p><p>The first thing is to have a Microservice. In my case, this will be a Spring Boot application with Actuator module, because we need a /<em>health</em> endpoint in this service.</p><p>Fortunately, Spring has a kind of “application generator” in <a href="http://start.spring.io/">start.spring.io</a>, so go there and generate an application with Spring Web and Spring Actuator modules and download it!</p><p>Now, run the maven package command to generate your FAT JAR:</p><pre>mvn package -DskipTests</pre><p>The ouput of this command will be a file called <strong>your-application.jar</strong>. In my case it is fargate-0.0.1-SNAPSHOT.jar.</p><p>The second step, is to dockerize my Spring Boot Application, and for this I’ll need a Dockerfile.</p><blockquote><em>I assume, that you are familiar with some Docker Concepts, like images. This is mandatory for using ECS, Fargate, and any other service that uses Docker containers.</em></blockquote><p>Dockerfile is the file that I’ll use to build the container image. It is like a cake recipe: a sequence of steps that Docker’s engine uses to generate images.</p><p>My image should be very simple, just with a Java Application, so the Dockerfile look like this:</p><pre>FROM openjdk:latestMAINTAINER <a href="/service/mailto:marcus.cavalcanti@b2wdigital.com">marcus.cavalcanti@gmail.com</a>WORKDIR /opt/springbootapp/ADD fargate-0.0.1-SNAPSHOT.jar /opt/springbootapp/RUN chmod +x fargate-0.0.1-SNAPSHOT.jarCMD [&quot;java&quot;, &quot;-jar&quot;, &quot;fargate-0.0.1-SNAPSHOT.jar&quot;]EXPOSE 8080</pre><p>Now, I can build the Dockerfile to generate the image. I’ll also tag this image. Using a tag is very important, because the tag will be the Microservice version to be deployed using Fargate.</p><p>To build the image, run the command below:</p><pre>docker build -t fargate-java:1.0.0 .</pre><p>You can see that I named (locally) my image as “fargate-java” and I used the tag “1.0.0”, the current version of my Microservice.</p><p>To check if the image was created succefully, run the command below:</p><pre>docker image list | grep fargate</pre><p>To check if the image is working properly, run the container with this command:</p><pre>docker run fargate-java:1.0.0</pre><p>Inside the container, make a curl in <em>localhost:8080/health</em>. If this command returns a 200 status code, it is working.</p><p>To work with Fargate properly, I must use Amazon Elastic Container Registry (ECR), which is a private <a href="https://docs.docker.com/registry/">Docker Registry</a> managed by AWS.</p><p>The container’s image that I’ll deploy using Fargate must already be in ECR, so let’s follow the steps below to accomplish this.</p><blockquote><em>I’ll assume that you already have aws-cli configured in you machine. If you don’t, use this link: </em><a href="https://docs.aws.amazon.com/cli/latest/reference/configure/"><em>https://docs.aws.amazon.com/cli/latest/reference/configure/</em></a></blockquote><p>To log in to ECR using your AWS Credentials, please run the command below:</p><pre>aws ecr get-login</pre><p>This command will generate an ouput. Take this output (<em>docker login</em> command) and run it as a command and voilà!</p><p>Now, I’ll tag the image that I created previously. I already have this image in my local machine, but this image is not in ECR, so let’s run this command:</p><pre>docker tag &lt;IMAGE ID LOCAL&gt; &lt;repositoryURI ECR&gt;:&lt;VERSION&gt;</pre><p>If you don’t know what the IMAGE ID is, run the command below and take the value of the third column, a hash value:</p><pre>docker image list | grep fargate</pre><p>In case you don’t know what your repository URI is in ECR, please run the command below, choosing your repository and taking the value of the “repositoryUri” field.</p><pre>aws ecr describe-repositories</pre><p>Your command should look like this:</p><pre>docker tag 3b893681f505 562821017172.dkr.ecr.us-east-1.amazonaws.com/marcus:1.0.0</pre><p>Now my image is already tagged in ECR and the final step is to push this image to AWS:</p><pre>docker push &lt;repositoryURI ECR&gt;</pre><p>Finally, with all prerequisites satisfied, it’s time to make my Microservice visible to the world.</p><p>Fargate is an abstraction of ECS, so instead of you managing your instances and clusters, it will accomplish this for you. In other words, Fargate is just a launch type of ECS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*k0ZDivEHg_jtUNKs.png" /></figure><p>ECS launch types: Fargate and EC2.</p><p>The first step is to define my container, in this case, as I created a Microservice from scratch, I must define a <strong>custom</strong> container. Click in <strong>configure </strong>and follow the steps below:</p><ol><li><strong>Container name</strong>: fargate-application;</li><li><strong>Image</strong>: the image path in ECR, including version/tag (e.g.: <em>562821017172.dkr.ecr.us-east-1.amazonaws.com/marcus:1.0.0</em>);</li><li><strong>Memory limits</strong>: define a soft-limit (128) to “tell” ECS about how much memory must be reserved for your container execution. Define a hard-limit (512) to kill your container when your application reaches this limit;</li><li><strong>Port mappings</strong>: my service is running on port :8080, so I mapped this port here.</li></ol><p>At this point, it is only necessary to define the fields listed below:</p><ol><li><strong>CPU Units: </strong>how many CPU Units does your application demand? In this case, how much CPU will your application share with other tasks in the same host machine. In case of use Fargate, you can leave this field blank.</li><li><strong>Entry point</strong>: sh, -c</li><li><strong>Command</strong>: /bin/sh -c “java -jar fargate-0.0.1-SNAPSHOT.jar”</li><li><strong>Working directory: </strong>/opt/springbootapp (same as used in Dockerfile)</li></ol><p>You can leave all the other fields in this section blank. Fargate will handle this for you.</p><blockquote><em>Technically I don’t need this because I created my Docker image with all required steps to run my Microservice.</em></blockquote><p>But if you need some specific tuning, or to mount a volume in your container, or configure a network, here is the place. You can also change all that information at another time with <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html">Task Definitions</a>.</p><p>With tasks I can configure how my containers will run. Like network configuration, memory, CPU, bind ports, container orchestration, etc.</p><p>As one of the purposes of using Fargate is just worry about application aspects, my first task will be very simple:</p><ol><li><strong>Task definition name</strong>: first-task-fargate;</li><li><strong>Network mode:</strong> <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html">awsvpc</a>;</li><li><strong>Task Execution Role: </strong>I created one previously with full ECS access;</li><li><strong>Compabilities</strong>: FARGATE;</li><li><strong>Task memory</strong>: 512mb (you’re billed for this);</li><li><strong>Task CPU</strong>: 0.25 vCPU (you’re billed for this).</li></ol><p>After configuring my custom container and also my first task, I want AWS to automatically create an ELB (ALB) for me. For this, I have to turn my container/task into a service.</p><ol><li><strong>Service name: </strong>automatically generated;</li><li><strong>Number os desired tasks</strong>: this is an IMPORTANT piece of information, this is about how many containers, or applications instances you’ll have. It is always a good idea to choose more than one, so choose 2;</li><li><strong>Security group: </strong>automatically create new;</li><li><strong>Load balancer type: </strong>Application Load Balancer;</li><li><strong>Load balancer listener port: </strong>the port that your application is running inside the container;</li><li><strong>Load balancer listener protocol</strong>: HTTP.</li></ol><p>An ECS Cluster is a logical organization to separate resources, like memory and CPU.</p><p>A cluster can have many instances and a given container (task) runs in a single instance of the cluster. A service, can run in many instances of the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P8RphEqhzPzDR1Vr.png" /></figure><p>Diagram of ECS objects and how they relate</p><p>In my case, I’ll create a new cluster, so follow these steps:</p><ol><li><strong>Cluster name</strong>: my-cluster;</li><li><strong>VPC ID</strong>: automatically create new;</li><li><strong>Subnets</strong>: automatically create new.</li></ol><p>The last step in the AWS console, is review all the previous steps.</p><p>If everything seems OK, click on the <strong>create</strong> button and wait for AWS to create all the components and integrations.</p><p>In case of success, you’ll see a screen like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gFIlNHugJkP7AoOh.png" /></figure><p>Now my Microservice is deployed and ready to receive HTTP calls. To discover which address I have to use to call this service, I entered in my ELB console and took the DNS name.</p><blockquote><em>Remembering that my service is running on port 8080 (default port of a SpringBoot application), if you may need to call this service through ELB on port 80, you can do this using target groups.</em></blockquote><p>Call the microservice with a simple cURL command like this:</p><pre>curl <a href="http://ecs-first-run-alb-1100347479.us-east-1.elb.amazonaws.com:8080/health">http://ecs-first-run-alb-1100347479.us-east-1.elb.amazonaws.com:8080/health</a></pre><p>This command should return this:</p><pre>{&quot;status&quot;:&quot;UP&quot;}</pre><p>When you use a cloud managed service like AWS Fargate, you’re using many building blocks (components) which are integrated with one another.</p><p>When I deployed this simple Microservice, under the hood AWS created a stack in Cloud Formation. With this stack, all the necessary resources were created as a single unit.</p><p>With Cloud Formation, AWS created a target group and also an Elastic Load Balance to expose our application as a HTTP resource.</p><p>Another element is when we create a VPC and subnets. We did this automatically with Fargate and it was accomplished using that Cloud Formation’s stack.</p><p>We’re also using Cloud Watch to send logs of all operations of each AWS component. In the same way, we’re using EC2 to manage ECS clusters and run tasks (containers).</p><p>Finally, we’re using IAM roles to grant access to our services and ECR to store our Docker image(s).</p><p>As you can see, is very simple and easy to deploy a Microservice using Fargate. For anyone are already working with containers, Fargate looks like a good option to replace Elastic Beanstalk, or even just EC2.</p><p>If your case is more complex, and demands other <em>scenarios</em> like features provided by a <a href="https://buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/">Service Mesh</a> solution and/or <a href="https://kubernetes.io/">Kubernetes</a>, you must start to work with containers. Containers are now the standard of industry for deploy and run applications.</p><p><em>Originally published at </em><a href="https://medium.com/@marcus.cavalcanti/deploying-microservices-with-aws-fargate-f0578d8199a3"><em>medium.com</em></a><em> on January 25, 2018.</em></p><h3><a href="https://medium.com/b2w-engineering?source=post_sidebar--------------------------post_sidebar-">B2W-Engineering</a></h3><h4>Somos a área de tecnologia e serviços digitais da B2W e…</h4><p>Following</p><h4>12</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3760820b5d09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/b2w-engineering-en/deploying-microservices-with-aws-fargate-3760820b5d09">Deploying Microservices with AWS Fargate</a> was originally published in <a href="https://medium.com/b2w-engineering-en">b2w engineering -en</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>