<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Zambrana on Medium]]></title>
        <description><![CDATA[Stories by David Zambrana on Medium]]></description>
        <link>https://medium.com/@zambranadavid87?source=rss-548e4f046c44------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*DZ-z6uMrh9EiF7WI.jpg</url>
            <title>Stories by David Zambrana on Medium</title>
            <link>https://medium.com/@zambranadavid87?source=rss-548e4f046c44------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:35:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zambranadavid87/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ramping up our test efforts with Cypress — part 2: Covering some special needs]]></title>
            <link>https://medium.com/coverwallet-engineering/ramping-up-our-test-efforts-with-cypress-part-2-covering-some-special-needs-187dfce48947?source=rss-548e4f046c44------2</link>
            <guid isPermaLink="false">https://medium.com/p/187dfce48947</guid>
            <category><![CDATA[cypress]]></category>
            <category><![CDATA[quality]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <dc:creator><![CDATA[David Zambrana]]></dc:creator>
            <pubDate>Fri, 21 Jan 2022 09:01:21 GMT</pubDate>
            <atom:updated>2022-01-21T09:01:21.689Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ramping up our test efforts with Cypress — part 2: Covering some special needs</h3><p>In the previous <a href="https://medium.com/coverwallet-engineering/ramping-up-our-test-efforts-with-cypress-part-1-the-setup-436d0b733d4b">post</a>, I wrote about how we at <em>Coverwallet</em> set up <em>Cypress</em> to keep up with many new features coming to our web applications. In this one, I want to share some interesting tweaks we built to cover a few edgy cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qq4YVp3fJpJvgIBi" /><figcaption>Photo by <a href="https://unsplash.com/@geekyshots?utm_source=medium&amp;utm_medium=referral">Geeky Shots</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Play 1: Intercepting carrier responses</h4><p><em>Coverwallet</em> is an online insurance broker. As such, we have to deal a lot with insurance carriers. This means that we need to develop and maintain integrations with them to provide our clients with insurance quotes.</p><p>Technically, we get data through our frontend app, pass it on to our backend and send it (secured) to the different insurance carriers to get estimations for the quotes that will be offered. We do that in more than one environment which complicates the work a little further.</p><p>When we started including the automatic tests in the test environment, such integrations ended up being a big pain. After a few iterations, we saw that we needed to change how we did things, and mocking those requests was the way to go.</p><p>First of all, we needed to create the fixtures as JSON files within the fixtures/ folder in the project. One per carrier and industry that we tested. The content of such JSON files is just the answers that we’d expect going through a happy path scenario.</p><p>Now we can proceed to our spec file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fw6fz7-yNQFZzayjhEEVGQ.png" /><figcaption>beforeEach hook in the spec file</figcaption></figure><p>We use the <a href="https://docs.cypress.io/api/commands/intercept">intercept</a> method to be able to set up the mock. There we indicate over which <em>XHRs</em> we want to use (only those containing /chubb/gl/), which fixture file we will be using (it has to coincide with the fixture file name) and we set the getChubbUw alias for it.</p><p>Now the test starts and when it gets to the point we need to use the mock, we leave it like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Nw3FChLrfsX5LZw0WQc-hg.png" /><figcaption>steps in the spec file</figcaption></figure><p>We just do a cy.wait along with our alias right at the moment that we expect the <em>XHR</em> to happen. As it already has information on what to respond with, it will automatically send the content of the JSON file passed on completing the mock.</p><p>Doing this trick in all the tests that needed to deal with carriers helped us to keep this test collection more stable and reliable.</p><h4>Play 2: Log me in!</h4><p>Another important feature of our product is the <em>Wallet</em> and the possibility for the user to log in and see what insurance products they have bought plus their being able to handle documents and requests. Access to the <em>Wallet</em> is allowed by providing the users with credentials once they have purchased a product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*25rKUsoI97jHYVkna-EY1A.png" /></figure><p>At some point, we decided to include the <strong>login</strong> tests in the regression suite that is meant to be run after every code push to the test environment. Historical data plus the number of features coming up around the <em>Wallet</em> revealed that we needed this too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*HAcL5riOKtoGaItPsuylEw.png" /></figure><p>We created those tests as independent spec files where we included different it for the happy path and the error scenarios.</p><p>Then we replicated this into every application mode we have (reusing some key pieces in separate js files). I am only showing two here.</p><p>So now we’re testing also the login in the pipeline where we run all those specs. Is that it? No, now comes a more interesting part.</p><p>We know that we can break the login from other parts of the platform, such as the <em>Identity provider</em> — a service that we use. As part of the project, we partnered with devs to try to understand the architecture and see where we should also run the tests from.</p><p>We found out there were 2 services that could impact the login functionality so we set up the tests to be run from there as well. We added a new job to our Circle-CI pipelines like the following using the Cypress orb:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*wGzb9Umq0BqvyXULfqPEQA.png" /><figcaption>config.yml file for Circle-CI</figcaption></figure><p>To clarify some data: We use an executor where we indicate which image it needs to run. We also set the login tag to have them separated in the Dashboard.</p><p>Especial mention to the spec entry: here we filter by the spec file(s) we want to run. As it is, it will find all the login specs within the project and leave the rest (because we don’t need to test other stuff from there).</p><p>For the other entries, we just require the deployment to be run as the trigger of the tests and we set the filter to only be run in the beta pipeline.</p><p>What if we wanted to run those login tests against production too? Easy. In your Cypress project, you can create a new <em>Cypress</em> <em>json</em> config file to be used for other envs, for example cypress.prod.json. In there, you leave configuration for your production environment such as the baseUrl you’re going to use. Then you’d need to create a new job in the pipeline which will be very similar to the previous one but with a couple of tweaks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*V34GCzllb2zZ5dgHveCEKQ.png" /><figcaption>config.yml</figcaption></figure><p>You will need to indicate your new config file. For the tag you may want a new one to have them separated in the Dashboard and then the filter for the pipeline branch.</p><p>This project was more DevOps-ish but it helped us understand more the application architecture plus we added some coverage that we definitely were missing. It paid off since day 1.</p><h4>Play 3: Authenticating through the backend</h4><p>This one is another example of how working closely with devs can boost the effectiveness of the test automation we create.</p><p>What we intend here is to use the backend to log in on our apps and forget about going through the UI every time we need to test something that requires us to be logged in. We are already testing the login in a different spec, so we don’t need to go through it again and again.</p><p>For it, we have created this piece.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgK2JDbw_WEowfM6C6kcaA.png" /><figcaption>bypassLogin function isolated in a js helper file</figcaption></figure><p>The getSSOToken is a function that runs a cy.request() against one of our services and grabs the token, to be used in the cy.visit().</p><p>With this, all we have to do in our spec file is to call the function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*Q7Z6Lrd_fNn8FKdBZg6MUQ.png" /><figcaption>beforeEach hook of our spec file</figcaption></figure><p>If the URL assertion passes, it will mean that we are logged in, so that will be our precondition for this test to start. If we don’t have it, it will mean that for some reason the login didn’t make it so we are not even interested in starting with the test.</p><p>For sure I could continue to share here more examples of interesting tweaks that we’ve worked on along with devs, but I am not willing to make this article endless. These are just a representation of how given a problem that we have — related to testing — we come up with a solution using available technology.</p><p>We also put a lot of focus on not just coming up with anything that solves the problem, but something that scales and works in the long term. Another point that I want to remark is how we work as a team to build these things and share knowledge. Very few pieces of this kind are built by individuals, there usually is a group of people behind them.</p><p>I hope you found this one interesting and helpful 💚</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=187dfce48947" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coverwallet-engineering/ramping-up-our-test-efforts-with-cypress-part-2-covering-some-special-needs-187dfce48947">Ramping up our test efforts with Cypress — part 2: Covering some special needs</a> was originally published in <a href="https://medium.com/coverwallet-engineering">Coverwallet Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ramping up our test efforts with Cypress —  part 1: The setup]]></title>
            <link>https://medium.com/coverwallet-engineering/ramping-up-our-test-efforts-with-cypress-part-1-the-setup-436d0b733d4b?source=rss-548e4f046c44------2</link>
            <guid isPermaLink="false">https://medium.com/p/436d0b733d4b</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[insurtech]]></category>
            <category><![CDATA[cypress]]></category>
            <dc:creator><![CDATA[David Zambrana]]></dc:creator>
            <pubDate>Fri, 18 Jun 2021 08:03:04 GMT</pubDate>
            <atom:updated>2021-06-18T08:03:04.894Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ramping up our test efforts with Cypress — part 1: The setup</h3><p>I decided to start writing a series of posts about our dealings with <em>Cypress</em> after some questions started to pour in, especially those coming from outsiders. In this one, I’ll surface how we set up the <em>Cypress</em> test framework to massively multiply the number of verifications we do on our apps while lowering our hands-on-it time in doing so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y3VbO3KXSVBwPyOt" /><figcaption>Photo by <a href="https://unsplash.com/@gronemo?utm_source=medium&amp;utm_medium=referral">Roméo A.</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>First, some context. <a href="https://www.coverwallet.com/about-us">Coverwallet</a> is an online platform that enables businesses to buy and manage a plethora of insurance products. We provide a friendly UI and UX to make it dead easy to bring businesses the best possible policy quotes based on the input we get from them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwqNLAb1Lk_5Jd3d7uW-uw.png" /><figcaption>Coverwallet quotes application</figcaption></figure><p>At the end of the form, we present the quotes so that they can be purchased, a service we also offer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cF_J2nKuNHnxubls0m3zdg.png" /><figcaption>Businesses can purchase quotes online or by phone</figcaption></figure><p>We started offering our services in the US alone but the story has recently changed. At the beginning of 2020 (those old good times), <a href="https://www.prnewswire.com/news-releases/aon-completes-acquisition-of-coverwallet-the-leading-digital-insurance-platform-for-small-and-medium-sized-businesses-300982331.html">we got acquired by Aon</a>, one of the biggest insurance brokers out there. Ever since then, the roadmap changed, and most of the new projects and product releases are those coming from within Aon, such as the <a href="https://www.prnewswire.com/news-releases/aon-launches-digital-cyber-insurance-for-small-and-middle-market-businesses-301183472.html">Cyber brokering solution</a> or <a href="https://www.prnewswire.co.uk/news-releases/aon-and-chubb-launch-new-digital-insurance-platform-in-france-for-small-and-medium-sized-businesses-866510099.html">offering our digital product in France</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mI70kJZEhvJH1Pl-McA7gg.png" /><figcaption>Aon France quotes app</figcaption></figure><p>I am sure you can feel a pattern here.</p><p>Aside from the visible things, there are several integration projects going on trying to ramp up operations, which in the end it is nothing but digitalizing processes.</p><p>With all this, our main focus now relies on scaling and improving our product to the point where it takes minimum effort to release it to the market. No need to mention that such a new product has to offer high-quality standards and, with the current speed we’re working at, that’s not feasible with functional testing alone.</p><h4>The approach</h4><p>We are divided into different <em>squads</em> taking care of the different parts of the application. That means that say, the <em>Checkout</em> tribe, takes care of all things related to the checkout process, from product definition, implementation, and delivery. The whole cycle remains within the team and it is the team’s purpose to have their backyard up and running.</p><p>We took a similar approach for testing the different apps and opted for <em>Cypress</em> as a game-changer since it is quite easy to set up and the learning curve is practically flat for a <em>Javascript</em> developer. We have independent repositories to store the test scenarios with custom configuration. In the beginning, we defined a series of e2e cases we needed to cover, basically some critical user flows, and went for them. For the sake of simplicity, let’s divide the scenarios between <em>Aon</em> and <em>Coverwallet</em> since the same app can be served in different modes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*b7hKP31RXEsCRKzB3fIWkg.png" /><figcaption>How spec files are distributed</figcaption></figure><p>Every spec can have from one to many tests. How we distribute them belongs to the test definition phase, since this can be handled in various ways.</p><p>Moving on into the configuration 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*Uw3AS4SaEt_Vz83fR4-ENw.png" /><figcaption>The project configuration file</figcaption></figure><p>This is an example of the cypress.json file where we set some high-level options that suit our needs. One of the nicest tricks we use here is the retries object, where we configure it to retry failed tests only when running headlessly, which relates to the CI run.</p><p>Now showing some of the scripts we use, both locally and on the CI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*KtpWUGXc9t_MHzq2MdnnDQ.png" /><figcaption>The scripts object in package.json</figcaption></figure><p>The open script raises the <em>Cypress</em> framework to run the specs in headed mode, being useful when creating new tests or debugging the ones that fail. We have added some DEBUG logs here to make it easier to debug the test scenarios.</p><p>The test script runs the whole collection in headless mode, used to verify that all is good after pushing any change. Now, the quick:test does pretty much the same but in a faster way, since we remove all the video processing between tests. Noteworthy, in case any of the tests fail here we might want to opt for open or test for verification.</p><p>Now, the test:cw and test:aon ones just run subsets of scenarios, focusing on the ones we care about for a specific moment.</p><p>We do use <a href="https://www.npmjs.com/package/husky">Husky</a> to improve our test code set as a <em>pre-commit</em> hook and configured as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*F419mD78C2OKRKIetF3csg.png" /><figcaption>Our Husky config</figcaption></figure><p>Moving on to the test scenarios, I will bring a few aspects since it is not the objective of this post to dive deep into this. We treat every it() as an e2e scenario, meaning that it does from one to many actions with a clear end that ought to be reflected in the test title. A couple of examples of this could be:</p><blockquote>it(‘should be able to complete a multipro app for a Charcuterie in Paris’)</blockquote><p>or</p><blockquote>it(‘should not be able to complete a cyber liability app for a beauty shop in Florida’)</blockquote><p>The goal is clear. For sure there are many other variables in here but since they do not alter the result for what we want to verify, we can reduce the verbosity to this.</p><p>As the different modes (<em>Coverwallet</em>, <em>Aon</em>) share many steps, we have bundled them into functions and isolated them into different files under support/ to later import them into the spec files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*v1Pb1HsxMjjDbe6HaLTl6Q.png" /><figcaption>cypress/support/utils/additional-store.js</figcaption></figure><p>Now we import it into the spec file and use it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*koGgYrZDGhp2incIQThPSg.png" /><figcaption>Spec file where we use the defined function as a step</figcaption></figure><h4>Continuous integration</h4><p>The test repositories have an action configured in a way that, whenever anyone merges new code intomaster, a new image of the test collection will be generated. This is a step that takes less than 3min and makes a <em>Docker</em> image available for other parties to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5z52gdFcg-3o7I7Ok5qZA.png" /><figcaption>What a completed action looks like on Github</figcaption></figure><p>Now let’s shift the focus to the development pipelines. At <em>Coverwallet</em> we use <a href="https://circleci.com/developer">Circle-ci</a> for our continuous integration, creating workflows that adapt to our needs and provides visibility to the team working on different development branches. I am going to obviate here the presence of some other jobs such as build or deploys and focus on the e2e test jobs.</p><p>Here, we need to take into account the total time for the workflow. We do not want huge delays between code pushes and formal verifications, so from our side we need to optimize as much as possible the duration of our e2e jobs. This belongs to the test definition stage as well as configuration tweaks to make them faster, such as finding the optimal pageLoadTimeOut for your app and trying to fail fast if that is the case.</p><p>To configure the test job we use <a href="https://circleci.com/developer/orbs/orb/cypress-io/cypress#quick-start">Cypress orbs</a> which provides a super simple and flexible template to set it up. It would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*YDaQQkga71lE0QxhBFMuEw.png" /><figcaption>.circle-ci/config.yml</figcaption></figure><p>A few things to remark from this example are:</p><ul><li>We use a executor previously defined with the information of the <em>Docker</em> image we want to use</li><li>We enable parallel runs with 2 machines giving it a group name and tagged as aon_us so that we can filter by this later on in the Cypress Dashboard</li><li>We filter by spec file as we did in our package.json though we could also specify a command to run instead of the spec key</li><li>This job only executes when the related deploy deploy_aon_us is successful and it only happens in beta</li></ul><p>If we do kind of the same for the other subset of scenarios, we now can run them separately and attach them to the corresponding deployments. This will happen every time new code is pushed to the beta branch of the app repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYKi1fuDBuEuwkYb2Ixlxg.png" /><figcaption>Example of a dev pipeline workflow. Yes, we are missing the EU tests here</figcaption></figure><p>Taking into account all the apps, we go to production around 150 times per week. It’s true that not all of these trigger the e2e tests since here I am including backend as well (these trigger other tests), but yet the number is high enough to have a robust verification process behind it.</p><p>As we have set the record feature activated, all the test runs will be recorded and pushed to the <em>Cypress Dashboard</em>, giving the team the possibility to watch the test runs and make real quick the feedback cycle. Once there, we can use the video or the screenshots to see what happened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8bJ47Uw3z8WwXzMXeq0zw.png" /><figcaption>Screenshot of an error in the test run</figcaption></figure><p>As of now, we are hovering the 20k test runs per month and increasing steadily as, usually, a few more tests are pushed in every sprint. One cannot practically notice the increase in the total figure sprint by sprint but she definitely can do when zooming out and seeing it from quarter to quarter.</p><p>We have also enabled the <em>Slack</em> integration in the <em>Dashboard</em> to push the results to different <em>Slack</em> channels to make sure that we don’t miss a thing.</p><h4>Wrapping thoughts</h4><p>This is the generic setup (special tweaks aside) we’ve been working with and for now, it is paying off well. Important to mention that this wasn’t just a QA project. Other parties such as <em>SRE</em> and <em>Dev </em>engineers<em> </em>also collaborated here. With this, we could involve more engineers in the quality process and help us make it bigger and more robust.</p><p>Is it perfect? Sure not, I believe there is nothing such as a unique and perfect way of approaching this kind of setups, but it definitely is the one that is helping us achieve our goals, getting quick and quality feedback in a way that lets us deploy faster and safer. There is still a lot to do and to learn from, but in the meantime, we are making it.</p><p>I hope you enjoyed this one, cheers ✍️</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=436d0b733d4b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coverwallet-engineering/ramping-up-our-test-efforts-with-cypress-part-1-the-setup-436d0b733d4b">Ramping up our test efforts with Cypress —  part 1: The setup</a> was originally published in <a href="https://medium.com/coverwallet-engineering">Coverwallet Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automation was all about an inner quality in the end]]></title>
            <link>https://medium.com/coverwallet-engineering/automation-was-all-about-an-inner-quality-in-the-end-b815f899a61e?source=rss-548e4f046c44------2</link>
            <guid isPermaLink="false">https://medium.com/p/b815f899a61e</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[test-automation]]></category>
            <dc:creator><![CDATA[David Zambrana]]></dc:creator>
            <pubDate>Fri, 04 Dec 2020 09:03:02 GMT</pubDate>
            <atom:updated>2021-07-16T08:18:07.076Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*ReHLCk876ayHiMx4cY1JSQ.png" /></figure><p>In this story, I want to share a little bit about what I do for a job that goes very much in line with a personal quality, or at least this is what I came to realize after some years in the field.</p><p>During these past years, I’ve been working as a <em>test automation engineer</em> — or any similar wording that you can think of. As it reads, I usually fall on the quality side of software development with a strong focus on automation. Thank goodness, in software, there is never-ending room for automation.</p><p>Like many QA automation fellows, I started as a — manual — QA engineer in a software consulting company. Of course, during the first months, you get to learn a lot; test plans, test coverage, testing cycles, equivalence classes, boundary value analysis… How to report a bug, and how not to. How to deal with the dev team and why it’s better to share a common space. I found it quite interesting, also because by then I wasn’t pretty much into coding and it sparked my curiosity. QA goes hand in hand with curiosity.</p><p>Yes, here comes the <strong>but</strong>. At some point, I found myself repeatedly running through long test plans, the infamous test cycles. So, we were doing these bi-weekly releases and we had to manually check the whole test suite like 2, 3, or 4 times per release…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*CEUXYme1ZnuXyWGyuZYUZQ.gif" /></figure><p>How did it go from all about curiosity to repeated cycles so fast? That certainly killed it. So my natural reaction was, I need to do something else, either to expand my current capabilities or move into a new field. It was then when I started reading and learning about test automation and it completely got me, especially because it sounded too well and it would kill the repetitive downside of it. When you see and try both options it is crystal clear how much value you add when you get your tests automated. Surely I knew about it when I started in the field, but by then it seemed so complex and distant that I didn’t even consider it. So I got hands on it.</p><p>📢 “What is this black magic?”</p><p>For sure I needed to catch up with some programming stuff I learned back at the university, aside from reading a lot of posts and going to tech talks. But it completely paid off, and then I started thinking, how much time did I lose at work testing things manually in repeated cycles? Of course, this is not every QA’s experience and thinking! If you manage to keep it on the curiosity side and away from repetitive tasks, one can perfectly go on with it and end up being a key player in a team. But it was not my case, and I’m happy with how it worked.</p><p>Also during my personal upgrade to automation, <em>CI/CD</em> systems were starting to be a real thing and that meant things were going to get harsh for manual testing. How would I be able to keep up with manual testing in <em>continuous integration</em> systems? The move had total sense.</p><p>📢 “Ok, bring it on!”</p><p>So first you focus on one language and try to master it. Never made it yet. I shifted from one language to another until I found one I’m comfortable with, and after a while, you realize that you don’t have to be a <em>rock star</em> as long as you keep pushing and providing value. Well… and learning, you have to be comfortable with that as well.</p><p>Also, frontend or backend? You get hands-on one approach, see all the upsides and downsides, then you try the other layer as well and compare… In my case, I don’t have a strong preference for any, but I do think that it is good to try both and have the whole perspective. Depending on how the application works and the team’s dynamics you might be more inclined to do more backend tests, or the other way round.</p><p>With time I also got more exposed to <em>CI/CD</em> systems and tried to make them part of my game, “to automate my test automations”. That sounded quite well and realized that SRE/DevOps colleagues are one of my best allies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*2Z0ODSKuc_fY9ulI_h5-RA.gif" /></figure><p>📢 “Ok now, this is cool…”</p><p>Viewing it in perspective, this has paid off so much and so well that I can’t see why one would reject it. It is true that you need to invest some time in the beginning, but once it is up and running it’s all benefits, and that includes the time you spend fixing failing tests.</p><p>This game is all about long-term investment, something that I particularly try to foster in any aspect of my life I am capable of: purchasing goods (cost vs expected use time) or learning material (courses, books, even tv shows…), mundane daily activities (vacuum cleaning delegated to a <em>Xiaomi</em> robot), investments (<a href="https://indexacapital.com/t/0YwkAy">robo-advisors for index funds</a> with worldwide presence)... Maybe I enjoy it because it goes hand in hand with my thinking? I have to recognize it too; I am a bit of a lazy person and get quite annoyed when I have to do repetitive tasks, but hey, that can be the trigger to force yourself out of it and focus on what you really want.</p><p>Sometimes I thought it was my job that drove me to be this picky in terms of time-saving in my personal life, but deep inside, I believe it is the other way round. If I lacked this thinking, I’d never had made the professional move. This was the trigger of all this. It can be true that now, the professional side spread to the personal one and a snow-ball effect started therein. But this snow-ball would never get any bigger if I didn’t grow this quality within me. My belief is that I share this with many colleagues that work on automation, not only speaking about testing. When you enjoy what you do in your job, or at least the philosophy underneath, at some point you realize that you have it present in your daily life.</p><p>Now, at <a href="https://www.coverwallet.com/jobs/tech"><em>Coverwallet</em></a> we try to push the same philosophy since our very aim is to simplify and ease the whole process of finding and providing insurance coverage for businesses using technology and an intuitive UI. When you realize how much time it can take for a business to find proper coverage and to get all the related documents, you see how much room there is for automation and I’m only speaking about the process here. Focusing on the test side alone, we’ve got quite some frontend applications, a myriad of backend services, third-party integrations with carriers, payment platforms, CRMs<em>, </em>data platforms, identity providers… All this means that we need a lot of automated tests to be able to track the status, using different approaches for every part of the application and the most difficult part of it, balancing it all and providing visibility to the tribes to be able to react quickly. It sounds fearsome and exciting at the same time, but if you share the view it can be a lot of fun too.</p><p>If you got until this point, thank you for reading! I hope you enjoyed it and I would like to hear other’s opinions on this. Meanwhile, I’m back to my testing stuff. During the time I took to write this post (writing time) several hundred tests run in different environments and I’ve got to have a look at some of the failing ones.</p><p>Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b815f899a61e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coverwallet-engineering/automation-was-all-about-an-inner-quality-in-the-end-b815f899a61e">Automation was all about an inner quality in the end</a> was originally published in <a href="https://medium.com/coverwallet-engineering">Coverwallet Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seeking the best test strategy]]></title>
            <link>https://medium.com/@zambranadavid87/seeking-the-best-test-strategy-4bdb3259eb91?source=rss-548e4f046c44------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bdb3259eb91</guid>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[qa]]></category>
            <dc:creator><![CDATA[David Zambrana]]></dc:creator>
            <pubDate>Sat, 23 May 2020 12:13:40 GMT</pubDate>
            <atom:updated>2020-05-23T12:13:40.665Z</atom:updated>
            <content:encoded><![CDATA[<p>It has been some years for me already into the quality assurance side of software products, and this involved working with quite different people, teams, organizations and industries -even though the industry does not make much difference when you just test software. During this time I came to learn some things worth sharing, which is the intention of this post.</p><p>Yet I acknowledge that I’ve got many more things to learn, not only professionally but in any aspect of life, something that makes everything fun and engaging and also makes yourself a little better in the end. If I ever get to a point where I think I’ve learned it all, I’d worry about myself (and pity those who spend time with me since I’d become such an arrogant individual).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*kmBDdaKmEiErweaywW_hpQ.png" /></figure><p>Going back to what I’ve got to learn so far takes me to the title of this post, <em>Seeking the best test strategy</em>. The truth is, there is not such thing like one best strategy or just one process for every team. No silver bullet. Or at least this is what I saw so far. If I ever discover it I might well end up removing this post. Having said that, sorry for the click-bait title 🙏.</p><p>Still, I think that what I share here might help those who try to look for it too. Actually, it’d had helped me a few years ago when I struggled to come up with something like this.</p><p>Sure there are some generic guidelines you can follow to help improve the quality loop in your teams. For instance, how do you handle the manual/automated balance? Should you go for an all-automated way or maybe you are happy with checking everything manually? How I see it, you need both of them.</p><h4>Automated &amp;&amp; manual</h4><p>Something that I got to understand -and experience- after working on different software products is that you need both in place. Of course this implies having the resources and time so I will assume that. This two activities test your product in very different ways and are complementary to each other. An automated test -a bot- will test it always in the same way, repeatedly. In no scenario the bot will get bored or forget a step. This is why it is so good to have a good automated regression suite because everything you check there is tested, period.</p><p>But, can we rely only on bots? Can you sleep well having only automated tests and not even checking by yourself if you can make it to the landing page for several days? Seriously, if you can -and your data tells you the same- do it, it will save you loads of time. In my case I did not find that yet. Surely, data and metrics can help a lot here, but I’ll dedicate another section to this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i8s67SNnfCZYRi2qxr3HjQ.jpeg" /></figure><p>Manual testing gives you the human side of this practice. What does that mean? As I got to understand, it gives you the alternative way. An individual, even trying to reproduce the same scenario can detour and get to the same point but using different steps or data. That is great to complement bots! Now, what if not one but many colleagues run manual tests on the application without any guideline set. There is just the application, different business flows, different clients to use and many individuals to test it -meaning different psychologies, assumptions, backgrounds, ways of interacting with the product-. Unpredictability makes its appearance, which is quite welcomed.</p><p>Of course this demands resources and time, but the beauty of this is that you don’t need to do it often. Depending on availability and results that this practices deliver you can plan to do this once every… week? month? Remember you’ve got the bots working 24/7 for you checking the critical things, and the bugs you find in this manual sessions are -or should be- less harmful and less likely to come back again.</p><p>Aside from this, a little bit of smoke testing every now and then by any dev or QA can help to provide peace of mind into the team.</p><h4>Remove noise &amp;&amp; focus</h4><p>One principle that I try to follow not only at work but in every aspect of life is this one, try to remove the noise, the things that don’t add value to be able to focus only on those things that do. Luckily, when you work in high productivity teams you see this a lot around you, but in case you don’t I encourage you to embrace it.</p><p>Let’s apply this to testing… What about that huge regression test suite without which you believe you cannot live but keeps you busy a few hours per week, which can translate to over the hundreds in a year. Is it worth it? Even if you think so, do you need all that checks there? Specially the ones giving you the FAIL status…</p><p>Or the continuous manual testing performed leaving the team no time to work on test automation or helping the engineers in other ways… Do you have data supporting that? That if your team instead of taking 10% of their time to automate things take 40% or 50%, everything will burn around you? If that is true, well you’ve got a problem. What happens when any of your mates goes on holidays or medical leave? Or when they leave for good? Any scenario of the kind would only make things worse, this is why you must be working on the steps to make your approach more scalable.</p><p>When you get to a point where you see that things are getting more stable without the need of being yourself always there, that is a good indicator! You’ve got time to help others and think of ways to improve the cycle which feeds the productivity loop you are interested in keeping ♻️</p><h4>Data &amp;&amp; metrics</h4><p>I am a true believer of data-driven decisions and if you don’t have data to support your arguments I cannot really take them wholly. Having said that I go back to our field where this applies as well.</p><p>For feeding such productivity loop, what data can you take to help to push there? Have you got metrics that track your site and warn you whenever anything goes wrong? If so, you are in a good position to keep pushing. Maybe you’re not tracking 100% of events but some of them, so use whatever you’ve got handy to work on. Such metrics provide a safety belt that can relieve QA engineers to stop manually testing such things and work on more productive stuff. Also having incident reports can help you a lot.</p><p>A while ago I attended an engineering talk about testing in production, I was curious about how any company would do it and it blew my mind. They completely removed all kinds of testing in pre-production environments -they even removed the environments since they were added noise!- and only focused on tracking metrics and acting upon them using Canary releases. Here there is a <a href="https://blog.getambassador.io/cloud-native-patterns-canary-release-1cb8f82d371a">nice post explaining it</a>. In my opinion this is brilliant, just thinking about all the time it could be saved just without having to maintain different environments… moving all that time to more productive activities.</p><h4>Experience</h4><p>Here we can differ between your whole experience as engineer/QA engineer and the experience in your current project. You should take advantage of both of them and use it to improve performance.</p><p>During your years as QA engineer you certainly learned a few things. Specially if you changed your employer or project, you saw that some issues usually happen in some kind of platforms, or in that other kind of processes this other bug is likely to come up… all that things add up to your experience and makes you valuable in a team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*Bp3rKPDGp-upE0u5BMdpFg.jpeg" /></figure><p>As of the experience in your actual project I want to focus on the specific technology components you work with. After some time you learn where the weak points are and where you should be more careful, so you might want to spend some more time there instead of on other parts or components where nothing usually happened. And yes… it could happen now and you’d have let a bug go into production, but thankfully there are processes to get quickly to the previous state. My point here is, play with the odds, learn where your system usually fails and take steps to amend it, and it’s ok to let loose other parts where you feel safer. Use your time wisely ⏳</p><p>To wrap all this up, I’d encourage you to try some of the points I stated if you are struggling with the quality process in place . There are quite different ways of tackling this and you need to find the one that works for you. I hope that this post has helped you somehow and in case you’ve got any feedback or objection I’d love to know about it. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bdb3259eb91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[QA @ Spotahome part 2: Testing our backend platform]]></title>
            <link>https://medium.com/spotahome-product/qa-spotahome-part-2-testing-our-backend-platform-907687c42fcf?source=rss-548e4f046c44------2</link>
            <guid isPermaLink="false">https://medium.com/p/907687c42fcf</guid>
            <category><![CDATA[gauge]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[David Zambrana]]></dc:creator>
            <pubDate>Tue, 17 Dec 2019 14:42:58 GMT</pubDate>
            <atom:updated>2019-12-17T14:42:58.627Z</atom:updated>
            <content:encoded><![CDATA[<h3>QA at Spotahome part 2: Testing our backend platform</h3><p>This is the second post of a series where we share how we test our applications and services at Spotahome. In the previous one we commented about our frontend testing framework and how we saw it to be escalable. In this second one we will focus on our backend test framework and the alignment we have with the existing test magic in other layers of the platform.</p><h3>Recalling the test pyramid</h3><p>Let’s remember the testing pyramid theory so that we don’t lose focus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HwssMHsUZBzB2g_g" /></figure><p>In the previous post, we talked about e2e tests, also known as UI tests. These are the ones that take more time to develop, more time to run and are as well the most sensitive ones since just small changes in the frontend or the backend of your application might break them.</p><p>Let’s focus now on integration tests which for us are those working at the API level. API tests are (or must be) faster to develop and to run since they do not require any user interface to interact with and they directly hit the different endpoints designed in your application.</p><p>Following the pyramid stratification, these shall be more numerous compared to UI e2e tests, so we can leverage on them to increase our test coverage.</p><h3>The platform behind</h3><p>Our backend platform follows the micro-service pattern, decomposing the application in small parts (services) that take care of one isolated piece of the whole business. We follow a hexagonal architecture (aka ports and adapters) to avoid having them highly coupled and facilitating testing activities. In case you are interested in this topic you can have a look at this <a href="https://fideloper.com/hexagonal-architecture">post</a>.</p><p>To expose all the endpoints we make use of APIs which are consumed by our BFFs using <em>GraphQL</em>.</p><p>Regarding the base code, it is written in <em>PHP</em> and it follows the <a href="https://www.thoughtworks.com/es/insights/blog/domain-driven-design-services-architecture">Domain driven design</a> (DDD) pattern.</p><p>With all this what we have in the end are different bounded contexts (<em>BCs</em>) that completely isolate each side of the business.</p><h3>The test framework</h3><p>To test this we opted for <a href="https://docs.gauge.org/overview.html"><strong><em>Gauge</em></strong></a> which is an open source tool to write and run tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*q7IrgjIELUDcpYn6" /></figure><p><em>Gauge</em> was created by <em>ThoughtWorks</em> (does this ring a bell?) and it is supported by a wide community now. It can be used in any platform and it supports many languages for writing your test code. Two key features that we liked are that it’s BDD and it supports data-driven executions, making your tests more escalable.</p><p>For writing the tests scenarios themselves <em>Gauge</em> uses specification and concept files. Basically, specification files are those containing the different scenarios written in a set of steps and concept files are just a collection of steps that you can reuse in the scenarios. You can read extensively about this in their <a href="https://docs.gauge.org/writing-specifications.html">docs</a>.</p><h4>Spec files</h4><pre>Create Booking Request</pre><pre>======================</pre><pre>Tags: booking</pre><pre>Create booking request and charge (OK)</pre><pre>- - - - - - - - - - - - - - - - - - - - - - - - - - ---------</pre><pre>* Login with &quot;test-user&quot;</pre><pre>* Create a booking lead for listing id &quot;999999&quot; with move in &quot;2021–01–01&quot; and move out &quot;2021–01–31&quot;</pre><pre>* Create a booking request from booking lead</pre><pre>* Charge booking</pre><pre>* Wait until the booking request is converted into booking</pre><pre>Create booking request for past dates (KO)</pre><pre>- - - - - - - - - - - - - - - - - - - - - - - - - - - ------------</pre><pre>* Login with &quot;test-user&quot;</pre><pre>* Create a booking lead for listing id &quot;999999&quot; with move in &quot;2002–01–01&quot; and move out &quot;2002–01–31&quot;</pre><pre>* Verify response status is &quot;422&quot;</pre><p>Pretty understandable, isn’t it? This one above is an example of a specification file written in <em>Markdown </em>language with two scenarios. The scenarios just describe what they do in different steps. Each one of these steps has test code behind, but they can also relate to a concept file since they can be a set of steps themselves.</p><p>Usually, at the project directory, you would have a specs/ folder where you put all this. Inside it, we have many other directories sorted by feature where we put the spec files and have them located.</p><p>Note that <em>Gauge</em> lets you tag your specifications and even your scenarios with the <em>Tag</em> keyword, being this a kind of labeling enabling you to search and filtering them when necessary.</p><h4>Concept files</h4><pre># Charge booking</pre><pre>* JSON from file &quot;jsonTemplate/bookings/chargeBookingRequest.json&quot; is set as variable &quot;chargeJson&quot;</pre><pre>* Set path &quot;/payment/booking-requests/{id}/chargeNow&quot; as name &quot;charge&quot; replacing path variable &quot;{id}&quot; with variable &quot;bookingRequest&quot;</pre><pre>* An HttpSampler with name &quot;chargeBR&quot; and path in variable &quot;charge&quot; and type &quot;PUT&quot; and port &quot;xxx&quot; and json &quot;chargeJson&quot;</pre><pre>* Response code is equal to &quot;204&quot;</pre><p>Here we show an example for the step “<em>Charge booking</em>” that we presented in our spec file. You see all the steps behind that one, meaning that every time we call “<em>Charge booking”</em> from a spec file, all these steps will run.</p><p>We usually do that when:</p><ol><li>There are a lot of small steps required for just one high level task</li><li>We do not want to show so much logic at the spec file level</li><li>We are going to reuse such step in many test scenarios</li></ol><h4>Implementation</h4><p>Let’s show as an example the implementation for the step <em>“An HttpSampler with name “chargeBR” and path in variable “charge” and type “PUT” and port “xxx” and JSON “chargeJson””</em></p><pre>@Step(&quot;An httpSampler with name &lt;name&gt; and path &lt;path&gt; and type &lt;type&gt; and port &lt;port&gt; and json &lt;json&gt;&quot;)</pre><pre>public void setHttpSamplerNameTypePortBodyGiven(String name, String path, String type, Integer port, String json) {</pre><pre>// Java code to send an http request with the passed on parameters</pre><pre>}</pre><p>Note the <em>@Step</em> annotation. It denotes that the function is a step used in the spec files. For the Java code itself, I am not going to enter in detail since we’ve got lots of libraries and toolings in place to make the implementation reusable. I just wanted to show how a step relates to a function here.</p><p>This code would live in a <em>Java</em> class along with many other steps being somehow related and sharing context. Some examples of these classes would be:</p><ul><li><em>BookingSteps</em></li><li><em>ListingsSteps</em></li><li><em>HttpRequestSteps</em></li><li><em>HttpResponseSteps</em></li><li><em>ManageDataRequests</em></li><li>…</li></ul><p>And so on it goes… The clear objective of this is to keep the steps neatly organized and easy to find (even though if you use a gauge plugin you can jump to them directly from the spec).</p><p>Wrapping all the implementation we have <em>JMeter</em> to be able to run load testing with the scenarios developed. We can configure it passing on some parameters for the number of threads, ramp-up and duration and see how our platform behaves under such load.</p><p>As commented, you can write your test code in different languages like <em>Java</em>, <em>Javascript</em>, <em>Python</em>, <em>C#</em> and <em>Golang</em>. In our case, we opted for <em>Java</em>. This decision was made a while ago. Had we been presented with that question again, we’d have probably opted for <em>Javascript</em> as most of our frameworks currently use it.</p><h4>Build</h4><p>To manage and build the project we use <em>Maven</em>. As many would know, <em>Maven</em> is an extensively used build automation tool from Apache used in <em>Java</em> projects, although it can also be used for other languages. We’ve been working with it from the very beginning and it’s been useful to us given the big tree of dependencies that we have to maintain, both in-house and external ones.</p><p>Since we’ve got all our API tests in the same place, they are all placed in the same repository. So after updating our test collection, be it adding new tests, removing or refactoring, the build is triggered just after pushing the changes to <em>master</em> branch to make sure all is fine. With that, we make available the latest version of the tests for the <em>BCs</em> to grab and run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/0*J0-Io8eNTcWbCGLX" /></figure><h3>Setting up the CI</h3><p>If you remember from our first post, we use <em>Brigade</em> + <em>GitItNow</em> as our CI system and our <em>BCs</em> use them in the same way. For every <em>BC</em>, we’ve got a <em>Brigade</em> file to indicate the events that must run in such pipeline.</p><p>With that, <em>GitItNow</em> (in-house developed magic frontend tool), makes quite easy to interact with the pipeline and brings visibility to the whole process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*HdYn3m-KXf7vklQM" /></figure><p>Above there’s one example of a pipeline for one of our BCs. As with our BFF pipelines, you can find a push event where we run some checks previous the deployment, a deploy event and a post deploy one were depending on the environment we may trigger different jobs and events.</p><p>For the <em>api-tests</em> event it gets the latest successful build from the test repository and runs it. In case anything went wrong, <em>GitItNow</em> makes available the logs for us to see the errors that might have happened. It also sends notifications to our Slack channels so that there’s no missing from our part and it enables us to answer swiftly.</p><p>Due to the number of tests in some BCs we opted to run them in parallel. To do this we need to be very careful with what to parallelize since we may be altering data used in other tests, so something to keep in mind when going for parallel runs.</p><p>What follows is a piece of Brigade code that enables parallel tests:</p><pre>async (brigadeEvent, project) =&gt; {</pre><pre>try {</pre><pre>return Promise.all([</pre><pre>apiTestsJob(brigadeEvent,project,&#39;search-dates&#39;,&#39;specs/search/dates).run(),</pre><pre>apiTestsJob(brigadeEvent,project,&#39;search-filters&#39;,&#39;specs/search/filters&#39;).run(),</pre><pre>apiTestsJob(brigadeEvent,project,&#39;search-no-filters&#39;,&#39;specs/search/no-filters).run(),</pre><pre>…</pre><p>In the end what they are is independent jobs triggered in an async way, and the event that wraps it up will return OK only if all the jobs did well.</p><p>What do we have in that <em>apiTestsJob</em>? Well, there it is some logic followed by the execution command to run the API tests with <em>gauge</em> on a given directory:</p><p>mvn -s &lt;settings&gt; gauge:execute -DspecsDir=${specsDir}</p><h3>Our strategy with API / Integration testing</h3><p>In the previous post, we closed it up commenting a little bit about our strategy and how we decide whether to extend or not the e2e collection depending on the case. I want to do the same with this one as a kind of wrap up.</p><p>As you might figure out the approach is not similar since they act upon different layers of the application, so the strategy cannot be the same. We know that the integration layer is less sensitive than the e2e one and they take less time to develop and run, so they can be a good ally if what we need is speed.</p><p>In the case when a new endpoint is released the actions are clear. We will develop API tests (assuming that the development of the endpoint already comes with unit tests) to put some coverage on it. Typically we will try to cover all the possible response codes (including the error ones) and decide if we need to test more logic on it, which we usually do.</p><p>On the other hand, if what we have is a high level feature coming from a new product requirement, here we have several options. Since these requirements usually come well documented and with an end-user perspective, we might be tempted to cover it with an e2e (UI) test, but that may not be the best option. Instead, we can create an integration test at the API level to cover such logic and avoid touching the UI components. It’s just a matter of what we need and creating the smoothest verification that validates the new feature.</p><p>So the question we try to ask ourselves now for every inquiry of this kind that comes along is, Do we really need an e2e test to cover this? Can we cover it with API tests?</p><p>Hey! I am not saying that we must avoid creating an e2e test. Maybe one of the requirements is that the end-user must see some UI component in the frontend or clicking on a button… If that’s the case, then let’s go for the e2e option. Otherwise, if we only need to check the business logic or if we see that the maintenance of the UI test will be too expensive, then we might be fine with the API one.</p><p>What we try to find with this exercise is a balance between speed and coverage and the optimization of testing efforts. This is no silver bullet and I am not saying that this is the way of doing things in every situation. One just needs to find their best approach. Here at Spotahome we are continuously iterating and trying to find the best way that works for us, in our current situation and with our resources.</p><p>To us, it’s quite interesting the output that this move has had. A year ago we were more bold with creating more tests at the UI level, which of course covered the functionality we wanted to release. In the midterm we saw that many of those tests created a lot of noise in the pipelines -false negatives. Why was that? As mentioned, UI tests are quite sensitive and they suffer at many changes pushed by the squads, being backend or frontend changes.</p><p>The result, many red pipelines and many hours lost trying to find out what was happening.</p><p>We analyzed all that and decided to move some of the test scenarios to the API level. Also, for the ones that really needed to be at the UI level, we applied some trimming on what we were validating -the minimum necessary.</p><p>The result: Now we see almost no noise in the pipelines, we got some extra speed developing more API tests, we still feel safe on the checks we have and no critical bug made its way to production… at least yet!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*WJvgspZ8vxaLmSGb" /></figure><p>So for the time being, we will continue pursuing this way since it’s demonstrated being pretty useful in our case.</p><p>This is what we wanted to share for this part #2. We hope you find it insightful, feedback is welcome! I want to thank you the colleagues/spotters that collaborated on the elaboration of this post since it is thanks to them why these posts bring some quality out there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=907687c42fcf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/qa-spotahome-part-2-testing-our-backend-platform-907687c42fcf">QA @ Spotahome part 2: Testing our backend platform</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[QA at [Spotahome] part 1: Testing our frontend platform]]></title>
            <link>https://medium.com/spotahome-product/qa-at-spotahome-part-1-testing-our-frontend-platform-ef3d3af28238?source=rss-548e4f046c44------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef3d3af28238</guid>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[cypress]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[David Zambrana]]></dc:creator>
            <pubDate>Tue, 11 Jun 2019 13:39:50 GMT</pubDate>
            <atom:updated>2019-06-11T19:57:03.650Z</atom:updated>
            <content:encoded><![CDATA[<h3>QA at Spotahome part 1: Testing our frontend platform</h3><p>This post is the first of a series where we will try to explain how we address Quality and test automation at Spotahome.</p><p>In this first one, I’m going to summarize the test automation we do on our frontend platforms, including the tools, the technology and strategies.</p><p>Our front-end architecture is built up based on the <a href="https://samnewman.io/patterns/architectural/bff">BFF approach</a> using Node + React, that combines the speed of JavaScript and uses new ways of rendering websites, making them highly dynamic and responsive to the users. For it, we make use of <a href="https://graphql.org/"><em>GraphQL</em></a> to request data to the backend. <em>GraphQL</em> was developed to cope with the need for more flexibility and efficiency that developers experience when interacting with REST APIs, making the workflow much more efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*yXa7cMaDCGVL-sdIQrlpYA.png" /></figure><h3>The test tool</h3><p>Little more than one year ago we decided to start using <a href="https://www.cypress.io/"><em>Cypress</em></a>, a tool that intends to overtake <em>Selenium</em> (or to make you forget about it right away), by ripping off all its setbacks such as the set up and debugging. If you want to dive deeper into this comparison, <a href="https://crossbrowsertesting.com/blog/test-automation/selenium-vs-cypress/">here</a> you can read further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VE_sTCyYs-ZjgqvPPJ3SA.png" /></figure><p>In their terms, <em>Cypress</em> is a testing tool built for the modern web. One can see lots of similarities among them, but the benefits and constraints you will face are quite different depending on which one you use. If what you’re seeking for is easiness of setting up and debugging, <em>Cypress</em> can be a pretty good match.</p><p>One big drawback in <em>Cypress</em> as of the time of this writing is cross-browser testing, since it can only run the tests in Chrome and Electron. Other browsers such as Firefox or Safari are on their roadmap but there’s no specific date to release them yet, so if you cannot go without this feature, you’ll need to wait for <em>Cypress</em> to include them in future versions.</p><p><em>Cypress</em> is an open source test runner that lets you write tests in JavaScript code. You can checkout their <a href="https://github.com/cypress-io/cypress">repository</a> and contribute to their mission if you feel so.</p><p>Despite its short life, you can find lots of references to it, be it <a href="https://www.youtube.com/watch?v=5FnalKRjpZk">lectures on <em>YouTube</em></a>, tech articles, Q/As on <em>Stack Overflow</em> or a <a href="https://gitter.im/cypress-io/cypress"><em>Gitter</em> chat</a>. It even appeared in the <strong><em>ThoughtWorks</em></strong> technology radar, adopt section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*7b_DMgJ2ZvpOsThrsHgttQ.png" /></figure><p>In our case, we made the shift from <em>Selenium</em> to <em>Cypress</em> for several reasons. The most important one, it seemed to be pretty easy to set up and work with, and that enabled us from QA to scale testing efforts by spreading its use among QAs and devs. Since it uses JavaScript, our dev team welcomed it and started to use it almost friction-less. Even having some weaknesses as commented above, the strong points were enough to outbalance them.</p><h3>The technology underneath</h3><p><em>Cypress</em> is written in JavaScript and lets you write tests in JavaScript. One thing to keep in mind is that <em>Cypress</em> is built up on top of <a href="http://mochajs.org/"><em>Mocha</em></a>, so if you are already used to write test scripts using <em>Mocha</em> as your test framework, you are likely to find some old friends here :)</p><p>From <em>Mocha</em>, <em>Cypress</em> adopts its BDD syntax which fits nicely with both integration and unit testing. Some of the syntax statements that you will find in this group are:</p><ul><li><em>describe</em></li><li><em>context</em></li><li><em>it</em></li><li><em>before</em></li><li><em>beforeEach</em></li><li><em>after</em></li><li><em>afterEach</em></li></ul><p>Another bundled technologies that <em>Cypress</em> comes with are <em>Chai</em>, <em>Chai-jQuery,</em> <em>Sinon</em>, and <em>Sinon-Chai</em>.</p><h3>How does a Cypress test look like</h3><p>As mentioned before, <em>Cypress</em> takes its syntax from <em>Mocha</em> to give shape to your test specs. The following is an example of a part of a test script that belongs to one of our suites:</p><pre><em>it(‘user should be able to see their profile info filled after saving their data in a booking request’, () =&gt; {cy.server();</em></pre><pre><em>cy.route(‘PUT’, ‘**/api/**/booking-request’).as(‘bookingRequest’);</em></pre><pre><em>cy.route(‘POST’, ‘**/api/formData’).as(‘personalData’);</em></pre><pre><em>cy.visit(</em></pre><pre><em>`/booking/${city}/${listingIdMadrid}/${checkInDate}/${checkOutDate}`);</em></pre><pre><em>cy.contains(‘.booking-form’, ‘Personal details’).should(‘be.visible’);</em></pre><pre><em>cy.get(‘[data-cy=”formName”]’).should(‘have.attr’, ‘value’).and(‘eq’, name);</em></pre><pre><em>cy.get(‘[data-cy=”formEmail”]’).should(‘have.attr’, ‘value’).and(‘eq’, email);</em></pre><pre><em>cy.get(‘[data-cy=”formPhone”]’).type(‘666777333’);</em></pre><pre><em>cy.get(‘[data-cy=”formBirthDay”]’).select(‘12’);</em></pre><pre><em>cy.get(‘[data-cy=”formBirthMonth”]’).select(‘3’);</em></pre><pre><em>cy.get(‘[data-cy=”formBirthYear”]’).select(‘1976’);</em></pre><pre><em>cy.get(‘[data-cy=”formGender”]’).select(‘male’);</em></pre><pre><em>cy.get(‘[data-cy=”formNationality”]’).select(‘AQ’);</em></pre><pre><em>cy.get(‘.booking-next-step’).should(‘be.enabled’).click();</em></pre><pre><em>cy.get(‘[data-cy=”formCouple”]’).select(‘1’);</em></pre><pre><em>cy.get(‘[data-cy=”message”]’).type(</em></pre><pre><em>‘hello there, I would like to reserve with Spotahome’</em></pre><pre><em>);</em></pre><pre><em>cy.get(‘[data-cy=”formOccupation”]’).select(‘professional’);</em></pre><pre><em>cy.get(‘.booking-additional-info-next-step’).click();</em></pre><pre><em>cy.wait(‘@personalData’);</em></pre><pre><em>cy.url().should(‘contain’, ‘payment’);</em></pre><pre><em>cy.contains(‘.braintree-option__label’, ‘Card’).click();</em></pre><pre><em>cardPayment(cardInfo.number, cardInfo.expiration, cardInfo.cvv);</em></pre><pre><em>cy.get(‘button[type=”submit”]’).click({force: true});</em></pre><pre><em>cy.wait(‘@bookingRequest’);</em></pre><pre><em>cy.url().should(‘contain’, ‘success’);</em></pre><pre>})</pre><p>When running the test in GUI mode, we see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9HEF5mbdqT3voILahcyrw.png" /></figure><p>What <em>Cypress</em> does is open a browser, then on the left you see the ‘live log’ which reflects everything the browser does during the test, and on the right you have the application under test.</p><p>Worth to mention here that <em>Cypress</em> is not an outsider tool that communicates with the browser through an API, but that it runs directly in the browser, and this makes it to be so close to the web application under test that you can do things that you can’t do in <em>Selenium</em>, such as stubbing DOM APIs.</p><p>One of the key features here is the <strong>debuggability</strong>. When the test finishes or fails, the state is not reset, meaning that you can hover around the ‘live log’ to see what or where was <em>Cypress</em> actually doing at that moment in time -a screenshot is displayed for that moment in time for every action. You can even open the browser console and get a full description of the steps you select on the log. This feature has boosted the whole process of test development so much that we can’t go without it now.</p><p>Let’s show it with an example: Let’s change some expected value in the test code to make it fail on purpose and run it afterwards:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RqkwDXcC_Y7L-oLYzWMEA.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F5VKbvrjxpVJCM%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F5VKbvrjxpVJCM%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExcjVjdzR1YjNuYXVyMnZtdzdlcWR1YmlzMDZuYTJ4MG4ybXR1OXUwOSZlcD12MV9naWZzX2dpZklkJmN0PWc%2F5VKbvrjxpVJCM%2Fgiphy.gif&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="348" frameborder="0" scrolling="no"><a href="https://medium.com/media/65b6e44ba9e7b9cb7450d4bf47648491/href">https://medium.com/media/65b6e44ba9e7b9cb7450d4bf47648491/href</a></iframe><p><em>Cypress</em> also lets you run the tests in headless mode. This way what you get is a command line log, with screenshots and video recordings of your tests.</p><h3>Setting up the CI</h3><p>For our CI system we use <a href="https://brigade.sh/"><em>Brigade</em></a><em>,</em> which is an event-based tool for running automated tasks on the cloud. Our colleagues from the DevOps team wrote a series of posts about how this works, so in case you are interested you can take a deep dive <a href="https://product.spotahome.com/a-tale-of-evolutionary-applications-iterations-and-performance-8053370c795f">here</a>.</p><p>As commented at the beginning of the post, we split our application in different BFFs, which lets us to build and deploy them independently, making the release process swifter and more easy to scale.</p><p>All of these BFFs have at least one e2e test repository attached, that will be triggered as a different event every time a pull request is merged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*GJlrX4W1FU5h2C8cm6nZaQ.png" /></figure><p>This case above illustrates the typical series of events that happen when merging a new Pull Request into our staging environment -BFF pipeline. First comes the <em>push </em>event (including unit tests 😍), then the deployment to the staging environment, and finally a <em>post_deploy_hook</em> triggers the configured e2e tests. We’ve got Slack integrated into this loop to notify us of every step and warn us in case anything went wrong.</p><p>The e2e tests are stored in a repository of their own (accounting to more than 15 repositories). We commit every change on e2e test code to those repositories to create a new Docker image that will be grabbed and executed by the BFF whenever the event comes.</p><p>Focusing on the e2e test event, the command we use to run the tests is:</p><p>$ <em>cypress run — record — group 4x-electron — parallel — ci-build-id ${buildId}</em></p><p>Briefly analyzing the params:</p><ul><li><em>record</em>: Sends the video recordings to Cypress <em>Dashboard</em> to keep track of the runs</li><li><em>group</em> &lt;name&gt;: Group recorded tests together under a single run</li><li><em>parallel</em>: Enables parallelization of the tests</li><li><em>buildId</em>: Necessary to defining a unique build or run</li></ul><p>Wait what? Sends the video recordings to a <em>dashboard?</em></p><h3>Cypress Dashboard</h3><p>The last part of the journey. <em>Cypress Dashboard</em> is a service that allows us to access to recorded tests, being pretty useful when we run them on the CI. It gives us information about all the test runs triggered, including logs, screenshots and videos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dn2jXkQrBvIuntqq_3VdFw.png" /></figure><p>In case we wanted to see what happened in our last failed test run, we can just open it and dive into the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EU5_OfkZNRHupj1jvXq7nQ.png" /></figure><p>Pictured above is an <strong>unusual</strong> test run where 18 ⚠️ tests failed, but it is in fact a good example to see how the dashboard displays information. Here, each row represents a different spec file, that might contain from 1 to ’n’ tests. All of them will be displayed in either green or red depending on their final state.</p><p>On the right column you can choose among the different options to check the run and dive into the errors, making available logs, screenshots and video recordings for that.</p><p>Even though we could go without the <em>Dashboard</em>, we found it pretty helpful and use it widely -we are on a paid plan that allows several of our engineers to have access to it and work in a more efficient way. The big win here is that we can be aware of the state of the application quite fast, and we can address any issue right away when anything fails. Once the failure is revealed and the potential impact calculated, we either fix the issue and test it again ASAP or we snooze the failing assertion until it gets fixed, so that we don’t introduce noise in the pipeline.</p><h3>A few words on strategy</h3><p>Last but not least I wanted to briefly share how we decide whether to automate or not any upcoming feature. I consider this an important topic and as such we try to get every member of the squad involved in that decision. It also depends on the squad’s feeling of the application and the given resources, so there’s not a correct answer to this.</p><p>In general terms, if we consider the new feature or change a critical one that can potentially impact in the business, we create one or more scenarios for it, depending on the acceptance criteria defined in the previous steps. This way we can be sure that such critical functionality is safe and protect ourselves against any code change that can affect it.</p><p>Otherwise, if we don’t consider that the new feature/change can have an impact in the business flow, we discuss if we want to create a new scenario for it, depending on the workload and the team’s availability. It can happen that we consider it’d be nice to have it tested, but it’s not that important, so we create the task and move it to the backlog, to grab it later when there is availability.</p><p>At Spotahome, besides the masters developing automated tests (devs included), there is the figure of the <em>functional tester</em>. They act basically as manual testers providing an extra effort on some defined scenarios for critical paths and checking other UI related aspects of the application for which we don’t have automatic tests. We do that because we are true believers that automated tests have a limit and cannot (and should not) cover everything. Embracing that human factor in quality has proved to be quite useful and we intend to continue to pursue that practice.</p><p>In the end, it is the main user story the one that holds all the acceptance criteria defined amongst PMs, tech leads and QA. From there, scenarios are created in different subtasks and the story is not completed until all the scenarios are created and running in our CI.</p><p>This is, in summary, the whole journey we’ve created to assure that our frontend platforms work as designed and meet the acceptance criteria set beforehand. Till we got here, we tested several approaches to decide which ones were the best for our needs.</p><p>Yet now, facing a huge increase in the number of engineers and reshaping the squads, are still seeing how we can take it to the next level, making every step as much automated as possible.</p><p>The actual figures that are in the game are:</p><p>10+ BFFs and increasing</p><p>150+ tests and increasing</p><p>130+ weekly test runs</p><p>As farewell, I want to say thank you to all the colleagues at Spotahome that collaborated on the elaboration of this post, and encourage the readers and QA guardians to give feedback and share their stories.</p><p>Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef3d3af28238" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spotahome-product/qa-at-spotahome-part-1-testing-our-frontend-platform-ef3d3af28238">QA at [Spotahome] part 1: Testing our frontend platform</a> was originally published in <a href="https://medium.com/spotahome-product">Spotahome Product</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>