<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gökhan KARAMAN on Medium]]></title>
        <description><![CDATA[Stories by Gökhan KARAMAN on Medium]]></description>
        <link>https://medium.com/@gokhanakaraman?source=rss-5992422ece30------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7-WDVhGK89p4bPwmfhK4AQ@2x.jpeg</url>
            <title>Stories by Gökhan KARAMAN on Medium</title>
            <link>https://medium.com/@gokhanakaraman?source=rss-5992422ece30------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:29:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gokhanakaraman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Don’t let your Flaky Tests ruin your Test Automation]]></title>
            <link>https://medium.com/@gokhanakaraman/dont-let-your-flaky-tests-ruin-your-test-automation-df2249745a9a?source=rss-5992422ece30------2</link>
            <guid isPermaLink="false">https://medium.com/p/df2249745a9a</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[ui-testing]]></category>
            <category><![CDATA[flaky-tests]]></category>
            <dc:creator><![CDATA[Gökhan KARAMAN]]></dc:creator>
            <pubDate>Sun, 12 Mar 2023 08:03:40 GMT</pubDate>
            <atom:updated>2023-03-12T08:03:40.370Z</atom:updated>
            <content:encoded><![CDATA[<p>Test automation is a crucial aspect of software development as it allows for efficient and consistent testing of the application’s functionality. However, one of the significant challenges in test automation is flakiness, which refers to the unexpected and inconsistent behavior of tests that were previously passing. This can lead to wasted time and resources trying to diagnose and fix flaky tests, and ultimately impact the development process and the quality of the final product.</p><h3><strong>What is exactly Flakiness in Test Automation?</strong></h3><p>Don’t you hate when the things are non-deterministic? as a Tester…Flakiness refers to the inconsistency of automated test results. Flaky tests are those that produce different results when run multiple times, even if the application hasn’t changed. Flakiness can be frustrating and time-consuming for testers, as they need to constantly re-run tests to ensure that they are working correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4JGfufEXU1rwh5Qk9aVuA.png" /></figure><h3>Causes of Flaky Tests</h3><ol><li><strong>Timing Issues</strong> Tests that rely on timing-sensitive elements such as waiting for a page to load or an element to appear, can become unreliable if the timing is not managed correctly. This can lead to flakiness in test automation.</li><li><strong>Test Environment Issues</strong> Specific configurations or data can also contribute to flakiness if not properly set up or maintained. For example, tests that run on different operating systems or browsers may behave differently, leading to flakiness.</li><li><strong>Poor Test Design</strong> Tests that lack atomicity, are overly broad or have too many dependencies can lead to unexpected test behavior. Parallelization of tests, or running multiple tests simultaneously, can also lead to resource contention and other issues that can cause tests to fail unexpectedly.</li><li><strong>Test Data Management</strong> Dependence on specific data sets that may change over time can lead to flaky tests. To avoid this, it’s important to implement techniques such as data isolation, data masking, and data virtualization.</li><li><strong>Lack of Proper Test Maintenance</strong> As codebase evolves, tests may become obsolete or no longer relevant, and they may start failing. To prevent this, it’s important to have a process in place for maintaining and updating tests regularly.</li><li><strong>Improper Test Reporting</strong> Incorrect or incomplete test reporting can make it difficult to diagnose and fix flaky tests. To prevent this, it’s important to use robust test reporting tools that provide detailed information about test execution, such as the test name, execution time, and the test result.</li></ol><h3>5 ways to avoid flaky tests</h3><ul><li>Use Robust and Reliable Test Automation Frameworks and Tools Choosing the right tools and frameworks for test automation can help avoid flakiness. It’s important to select tools that are well-maintained, have a large community of users, and are regularly updated to fix bugs and improve performance.</li><li>Properly Manage Test Environments and Configurations Properly setting up and maintaining test environments can help avoid flakiness. This includes ensuring that tests run on the same configurations, and that test environments are regularly updated and maintained.</li><li>Use Best Practices for Test Design Following best practices for test design, such as keeping tests atomic and avoiding unnecessary dependencies, can help avoid flakiness.</li><li>Regularly Maintain and Update Tests Regularly reviewing and updating tests can help catch any issues that may have been introduced over time, and prevent flakiness.</li><li>Use Robust Test Reporting Tools Using robust test reporting tools that provide detailed information about test execution can help diagnose and fix flaky tests. Monitoring test results and analyzing patterns in test failures can help identify tests that are particularly prone to flakiness. This information can then be used to update the tests or address underlying issues that are contributing to the instability.</li></ul><p>Flakiness in test automation can be caused by a variety of factors such as timing issues, test environment issues, poor test design, test parallelization, test data management, lack of proper test maintenance, and improper test reporting.In a nutshell, you can’t win the fight of flaky tests so there should be a level of awareness in the teams about flaky tests and should be part of the team culture to guard the automation process. To ensure a high-quality product and minimize the impact of flaky tests on the development process, it’s important to use robust tools, manage test environments and test data, follow best practices for test design, maintain and update tests regularly, and use robust test reporting tools.</p><p><strong>Thanks for reading and feel free to reach out :)</strong></p><p><strong><em>Cheers</em></strong>🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df2249745a9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Game Changer IDE from JetBrains — AQUA]]></title>
            <link>https://medium.com/@gokhanakaraman/new-game-changer-ide-from-jetbrains-aqua-7629623d8cf9?source=rss-5992422ece30------2</link>
            <guid isPermaLink="false">https://medium.com/p/7629623d8cf9</guid>
            <category><![CDATA[ui-automation-testing]]></category>
            <category><![CDATA[jetbrains]]></category>
            <category><![CDATA[test-automation]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[aqua]]></category>
            <dc:creator><![CDATA[Gökhan KARAMAN]]></dc:creator>
            <pubDate>Fri, 30 Dec 2022 17:25:33 GMT</pubDate>
            <atom:updated>2023-01-02T09:41:02.731Z</atom:updated>
            <content:encoded><![CDATA[<p>Finally, there is a proper IDE dedicated to automation in the market to code easily and effectively for test engineers. It has a lot of good features to help us code easily for automation projects.</p><p>In my opinion, the first priority of a test engineer should be bring the testing perspective into the product. So They should focus more on delivering the product with the all expected requirements. Because that is what matters in the end. In order to do that, We should be fast in delivering on time. So Aqua brings more functionality as a built-in to save us time. From now on, we don’t need to deal with the parts that have been developed many times before with their successful variants in them.</p><blockquote>“A powerful IDE for test automation” JetBrains</blockquote><p><em>There is an option to have Aqua </em><a href="https://plugins.jetbrains.com/plugin/20175-aqua"><strong><em>as a plugin</em></strong></a><em> and it is compatible with your current IDEs such as IntelliJ IDEA Ultimate, AppCode, Aqua, CLion, GoLand, PhpStorm, PyCharm Professional, Rider, RubyMine, and WebStorm. You can still use the features from Aqua If you don&#39;t want to download any other new IDE to your computer.</em></p><h4>What does the Aqua bring to the table?</h4><p><strong>1- Page Object Model Support</strong></p><p>It has been popular thing to use Page Object Design Model in test automation projects because that is the most fitting model to the requirements of testing and the advantage of the model is that it reduces code duplication and improves test maintenance. So there is an option to auto-create a Page-Object with a Page Factory design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*lGk700HqaO2VCaVFdY4WNA.png" /><figcaption><a href="https://www.jetbrains.com/help/idea/selenium-page-objects.html">Creating a new selenium page object</a></figcaption></figure><p><strong>2- Supports 3 languages</strong></p><p>Aqua is lately supporting to development of UI Automations in <strong>Java, Kotlin, and Python </strong>but it has a good highlighting and validation of XPath, CSS, and JavaScript in UI testing APIs for Java/Kotlin/Groovy/Python so I suppose there will be more in the future.</p><p><strong>3- Powerful coding</strong></p><p>I am seeing some code snippets as an example when I try to create a new client. So Aqua allows us to spend our valuable time on the testing approach and saves us time. Thence it has the usual powerful coding features, auto-complete, code snippets, etc.</p><p><strong>4- Built-in Web Inspector</strong></p><p>with integrated browser and CSS/XPath locator evaluator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Pnu8QQn-mcx1RdfiIBIHA.png" /></figure><p>If you have a big screen, you don’t need to go to your browser to inspect your elements. Aqua has a built-in web inspector so you can easily locate your elements without leaving your code screen.</p><p><strong>5- Proper IDE for automation scripts</strong></p><p>I would say it is totally focused on scripting/<strong>debugging</strong> for automated test scripts.</p><ul><li>Code completion for element locators. Verified CSS selectors and XPath expressions suggested based on the website open in Web Inspector</li><li>Code completion for HTML tags, attributes, and CSS properties in tests.</li><li>Run Selenoid action in browsers.json</li><li>Issue trackers for Allure Reporting and Serenity BDD</li><li>Code completion and validation for configuration files of Selenium Grid, Selenoid, and Allure Framework</li><li>Line markers with quick navigation to @Step usages in code.</li><li><strong>Find usages</strong> and code completion for @Feature, @Epic, @Story, @Owner, and @Lead usages for Allure Framework.</li></ul><p><strong>6- Easy to adopt for the companies</strong></p><p>It has been a long time since most of the companies are already using JetBrains products so would be easy to use this familiar IDE for everybody on the team. I think JetBrains is even pointing to the developers so they should write automation scripts since we don&#39;t have enough technical people in Test.</p><p><strong>7- So easy to create a new test project from scratch</strong></p><p>You could do this on your first day on the new job and you are ready to develop test cases for your new product. When you open the Aqua, click on the New -&gt; Project</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ydOK-25Hyy7lfk9pYzoRIw.png" /></figure><p>after choosing Selenium, I can see the options for languages, build, framework even JDK variants.</p><ul><li>Integrated reporting tool and assertions</li></ul><p>This part is amazing. No need to implement reporting or assertions manually anymore! We also have the option to select the selenium versions. Once the selenium dependencies are loaded it shows the suggestion to download the selenium UI Testing Library plugin from the Marketplace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X4Wb6_SeJUVUMTe08hSviA.png" /><figcaption>Creating a New Project</figcaption></figure><ul><li>No need to configure a test runner</li></ul><p>When you create the project with the selected options described in the picture above, you can directly run the test from the main class. The IDE will index and download the dependencies first, then run the example test. This could take a couple of mins. But the result is amazing;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzYs571dr7DAaV0o4P8qbw.png" /><figcaption>The first screen of the new project</figcaption></figure><ul><li>Easily filter/locate and create your elements with Locators Evaluators</li></ul><p>This part is also worth talking about it. You can create your Page Object Elements just by clicking the + button from Locator’s window and also choosing the type of the locators.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1VMLJPWrcDV30Sb85a2Rvw.png" /><figcaption>Locators in Web Inspector</figcaption></figure><ul><li>So easy to make API calls</li></ul><p>When you click File -&gt; New -&gt; HTTP Request, IDE generates an HTTP client for you and directly links an env. No need to configure any runner, no need to even search/think of any code for making API requests, Aqua offers code snippets in the Examples</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L27uk_WaCCHqYPmworhnUg.png" /><figcaption>http-client</figcaption></figure><ul><li>Docker</li></ul><p>Aqua has a Services tab as a built-in like the other JetBrains IDEs. So you can easily orchestrate your dockers from here. It allows us to run all docker functionalities from this tab such as build/debug/run/monitor the images</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Bg56LHi1OCzbLFcYG02Ng.png" /><figcaption>Docker</figcaption></figure><p>I see some <strong>cons</strong> in Aqua but I hope it will have more features in the future as far as JetBrains has the trust of developers. For the <strong>flexibility</strong> and <strong>reliability</strong> of the testing, We need to develop/run our test scripts in multiple browsers. So Aqua has only Selenium support for now.</p><p>I personally thank JetBrains. It has been a need for a long time to make the Test Engineers build quickly in a dedicated IDE. I would expect the same solution for the Mobile UI Automation projects.</p><p><strong>Thanks for reading and feel free to reach out :)</strong></p><p><strong><em>Cheers</em></strong>🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7629623d8cf9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Karate Framework vs Rest Assured]]></title>
            <link>https://medium.com/getir/karate-framework-vs-rest-assured-95482a61002e?source=rss-5992422ece30------2</link>
            <guid isPermaLink="false">https://medium.com/p/95482a61002e</guid>
            <category><![CDATA[restassured]]></category>
            <category><![CDATA[api-test-automation]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[karate-framework]]></category>
            <category><![CDATA[restassured-java]]></category>
            <dc:creator><![CDATA[Gökhan KARAMAN]]></dc:creator>
            <pubDate>Mon, 04 Apr 2022 09:04:50 GMT</pubDate>
            <atom:updated>2022-04-04T09:11:47.137Z</atom:updated>
            <content:encoded><![CDATA[<p>If you want to have a stable application you need to spend a notable amount of time on testing and making sure that every business feature meets the defined requirements. Sometimes the whole testing process is one of the time-consuming parts of a software delivery cycle. It is also known that automation reduces time spent on repeated actions and eventually helps to reduce LTTC (lead time to change). That’s why we always should strive to use automation for such kinds of processes. Automation tools allow us to save time, provide means to reduce the time spent on repetitive manual tasks, shift quality left, and much more.</p><h4>Do we really need API testing?</h4><p>Since the applications are moved to the cloud, the importance of the APIs has been critical. API testing is a must with the enhancement of the microservice architecture and being mediator platforms. Many features of the applications that we use daily rely on hundreds of different synergistic APIs on the microservice architectures, if one of the services fails then we will have limited operational capacity of the related service.</p><h4><strong>What are the 4 most important criteria for finding the optimum API testing tools?</strong></h4><ul><li>Ease-of-use and Capabilities</li><li>Ability to optimize your workflows</li><li>Automation</li><li>Management &amp; Maintenance</li></ul><p>The ideal API testing tool needs to be easy to use for a beginner user to pick up and start using with minimal training. It should include capabilities according to your needs. For example, if you are going to execute data-driven tests, the framework should have support for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQ7M6Q8ut65a5Rkr2dm8kw.jpeg" /><figcaption><strong>Some specific key capabilities the framework should have</strong></figcaption></figure><p>Among many other qualities, your API testing framework must provide the ability to optimize your business flow. It should comprise intelligent mechanisms that simplify things you do daily. Think of test creation, monitoring, working with test data, and integrating your test solution into your requirements of the business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0S-7s661_UeqMYUroZEfOQ.jpeg" /><figcaption><strong>Abilities to optimize your requirements</strong></figcaption></figure><p>It’s valuable for your API testing tool to provide a mechanism to run your tests in automation. As we can imagine, manually testing APIs is not realistic, reliable, or simple. There are usually several endpoints in the microservice architectures and it is important to automate them. Thus we can programmatically trigger our test suites as a part of the CI / CD process.</p><p>In this article, I will compare two of the most popular API test automation tools, Karate Framework and Rest Assured, based on my experience.</p><p><strong>Karate</strong> was created by Intuit in April of 2017 and it gained popularity pretty fast. Developed and maintained by <a href="https://www.linkedin.com/in/ptrthomas/">Peter Thomas</a> who created a new company called <a href="https://www.linkedin.com/company/karatelabs/">KarateLabs</a> and moved the Karate framework support to the new company. Thanks to him test automation was made simple. It currently gets about 5.6k stars on GitHub. That’s quite impressive for a tool that is around for 5 years. The latest released version is 1.1.0.</p><p><em>Karate Automation Testing Framework was based on Cucumber JVM but as of version 0.9.0, not anymore(</em><a href="https://github.com/intuit/karate/issues/444#issuecomment-419852761"><em>see why</em></a><em>) but retains the syntax.</em></p><blockquote>Karate is the only open-source tool to combine API test-automation, <a href="https://github.com/karatelabs/karate/blob/master/karate-netty">mocks</a>,<a href="https://github.com/karatelabs/karate/blob/master/karate-gatling"> performance-testing</a> and even <a href="https://github.com/karatelabs/karate/blob/master/karate-core">UI automation</a> into a single, <em>unified</em> framework. The BDD syntax popularized by Cucumber is language-neutral, and easy for non-programmers. Assertions and HTML reports are built-in, and you can run tests in parallel to achieve high speed of execution.</blockquote><p>There are several articles and videos about how to create a project in Karate or how to implement DSL (Domain Specific Language) methods in Karate so you can easily learn how to do it. Here, I want to talk about how easy it is to develop tests in this framework and what you should be aware of. If you have already read <a href="https://medium.com/getir/java-lang-outofmemoryerror-java-heap-space-in-karate-framework-dc5ad83fcd1b">my last article</a> about this framework and you have a large number of tests, you should be careful in choosing this framework. It can enchant you easily with powerful skills but when it comes to performance, you should check if the memory error appears in the latest version. I have been having a memory problem with the karate framework in versions before v1.1.0.RC1.</p><h4>So let’s dig a little deeper</h4><p>A karate code looks like the one below;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*K6AXmb-gmKrCe6LhZG1j-w.png" /></figure><p>Tests are written in Gherkin format which is also used for BDD, but <a href="https://stackoverflow.com/questions/47795762/in-karate-how-we-can-collaboratively-work-along-with-ba-to-automate-business-sce/47799207#47799207">Karate is not true BDD</a>. There are several API request methods which are PUT, POST, DELETE, GET. In another language or framework, you need to declare those methods with a library or you need to develop the methods, while in karate making a request is very simple. Just write the type of request that you need after the “method” keyword.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*jx0ttGERpUzWOUQAHMe7Jg.png" /></figure><p>It is a <strong>keyword-driven framework</strong> so you don’t need to declare those methods which allow you to only focus on writing your tests. There are several benefits of this like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gsnjy3OpwR9HBfKh1zpHtQ.png" /><figcaption>reading files with <strong>call or callonce</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhqI-Jz7WxpYEWQp98ihJw.png" /><figcaption>assertion and validation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*3HwcsnOoiMNsfvStgEM5bA.png" /><figcaption>parsing response directly</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*d8Jq2Zd1CUG4hmbiFXH2og.png" /><figcaption>mapping header, body and param</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*XIgFfzQ6d1Mb7iWm2fW9JA.png" /><figcaption>manipulating JSON with <strong>replace</strong></figcaption></figure><h4>Data-Driven tests are so easy:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAXjYFuCq5_QutsUS82xWA.png" /></figure><h4>call vs. callonce</h4><p>If you use the <strong>call</strong> keyword in Background, your call method will be compiled for each scenario in that feature file. If you use the <strong>callonce</strong> keyword, your method is going to be compiled <strong>once</strong> and <strong>cached</strong> for each scenario instead of executing that method (or feature) again and again. Let’s have a look at it;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ofeX5DT7GwVmk-3aqKJBg.png" /></figure><p><strong>I have observed that callonce is making the test a bit slower </strong>because it caches the response of the request and reads from the cache for each scenario so if your response is large, it is going to make your tests slower. <strong>Depending on your scenario,</strong> <strong>you may prefer making that request every time for each case with the call keyword instead of caching with callonce.</strong></p><p><strong>REST Assured</strong> is developed and maintained by <a href="https://www.linkedin.com/in/johan-haleby-a1163b4a/">Johan Haleby</a>. Johan started the project when he was working at <a href="https://www.jayway.com/">Jayway</a> back in December of 2010. The project is now sponsored by <a href="https://www.linkedin.com/company/parkster/">Parkster</a>. The latest version is 4.5.0.</p><p>If you only need to check requests and responses, this library will simplify things. In REST Assured, you don’t have to create boilerplate code to test and validate complex REST responses. You can also use modeling your request and responses since Rest Assured framework is based on java.</p><p>Rest Assured framework has a BDD or Gherkin support that helps to enhance the readability of the code and clean coding. REST Assured assists in easy integration of classic testing frameworks such as JUnit and TestNG.</p><p>A rest-assured code looks like the one below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrqZTt__dHKGcKF5Ow4awA.png" /></figure><p>Data-driven tests are easy but you need to implement dataProvider from TestNG or equivalent library:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gLd2dr2lg-ckEK9k0QUTBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*TjSLmxsiQQ4Iinftuu-EKQ.jpeg" /></figure><h3>Comparing Test Execution Times</h3><p>I have observed that Rest Assured is a bit slower than Karate when it comes to executing tests. It can cost you time when you have large amounts of test packages. I have made a small example to show this and used the same libraries in both project and framework’s libraries have almost the same lines of code. Rest assured has 46k lines of code and Karate has 50k. I used the same API to test and API was stable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_0D7cUfb-4x_nFeMf3oBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kjws1Oj6hFrYKPny9xyNiw.png" /><figcaption>Test Execution Times <strong>Karate vs Rest Assured</strong></figcaption></figure><p>In my opinion, Karate Automation Test Framework makes more sense, especially for the teams that create and maintain web services and where acceptance testing is not the highest priority. Karate is not for the teams that are willing to perform BDD even if it looks like that.</p><p><strong>References</strong></p><ol><li><a href="https://github.com/karatelabs/karate">Github — KarateLabs</a></li><li><a href="https://rest-assured.io/">Rest Assured</a></li><li><a href="https://pdfroom.com/books/the-cucumber-for-java-book/EBgjel4adoN">Cucumber for Java</a></li></ol><p>In general, I share my experiences and the work that we think is cool in the Getir Core Automation Team. I am thrilled to be a part of the Getir technology team and very excited about the work we will do in the future. If you would like to be a part of this team, please do not hesitate to apply to one of our open positions at <a href="http://career.getir.com/">career.getir.com</a>. 😊</p><p><strong>Thanks for reading and feel free to reach out :)</strong></p><p><strong><em>Cheers</em></strong>🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95482a61002e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/getir/karate-framework-vs-rest-assured-95482a61002e">Karate Framework vs Rest Assured</a> was originally published in <a href="https://medium.com/getir">Getir</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“java.lang.OutOfMemoryError: Java heap space” in Karate Framework]]></title>
            <link>https://medium.com/getir/java-lang-outofmemoryerror-java-heap-space-in-karate-framework-dc5ad83fcd1b?source=rss-5992422ece30------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc5ad83fcd1b</guid>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[outofmemoryerror]]></category>
            <category><![CDATA[api-test]]></category>
            <category><![CDATA[karate-framework]]></category>
            <category><![CDATA[api-test-automation]]></category>
            <dc:creator><![CDATA[Gökhan KARAMAN]]></dc:creator>
            <pubDate>Wed, 26 Jan 2022 16:47:16 GMT</pubDate>
            <atom:updated>2022-01-31T08:12:41.184Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s technology, the vast majority of applications are served by backend services, so testing the backend is one of the important parts of your product quality lifecycle. You need to develop test cases for each endpoint, especially if your company is using a micro-service architecture.</p><p>As you might agree, backend tests should be easy to implement. Considering the popularity of the <a href="https://github.com/karatelabs/karate">Karate automation testing framework</a>, ease of developing cases and low coding requirements, we have chosen it as a platform for automation of our backend tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cG_XVaVWKLqAel3u.png" /><figcaption>Image on <a href="https://en.wikipedia.org/wiki/Karate_(software)">Wikipedia</a></figcaption></figure><p><strong>If you need to develop a significant amount of test cases and run them in a single run, you may have discovered that this framework has some problems and consumes excessive memory. </strong>You get a <strong>“</strong><a href="https://github.com/karatelabs/karate/issues?q=is%3Aissue+OutOfMemoryError+is%3Aclosed"><strong>java.lang.OutOfMemoryError: Java heap space</strong></a><strong>” </strong>error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xlmOaT7kVLOQ5jRYY5YjxQ.png" /><figcaption>Screenshot of the error</figcaption></figure><p>This error occurs when the tests are over a certain limit (I can’t specify the exact number). For example, we have a test package with 203 scenarios that run fine, but the other package has 376 scenarios and it’s getting this error.</p><p>When we dive deeper into the investigation of an error, we see that the framework has a problem with <strong>cleaning the heap space</strong>. Even if we increase the heap size to the maximum, we’re still getting the error. We have run the test package which returns this error on an m5.2xlarge EC2 instance with the 8 CPUs and 32GiB memory. The results<em>, </em>unfortunately, were the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*lcFjZnVQzwgECLb6e_9alw.png" /></figure><p>We have disabled all the logs, minimized the dependencies in pom.xml and monitored the memory usage of the tests again. We found that the CPU usage was around 350% while running only one scenario.</p><p>Imagine that you have already developed more than 3,000 test cases, but they aren’t working. What should you do? To solve the issue in a timely manner and support aggressive delivery timelines, we focused on finding a way to run existing tests rather than investing in the reimplementation of the same tests with another framework or even in another language.</p><h3>How did we run those Karate tests?</h3><p>We realized that the problem is running many suites together. So if we create <strong>an architecture that runs tests case by case</strong>, we might find our way around the issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*duA60wNm51xy9Bi4wpOzQA.png" /><figcaption>The architecture</figcaption></figure><p>Here is a running karate test command with maven.</p><pre>mvn clean test -Dtest=TestRunner -DfailIfNoTests=false -DstoppingStep=null &#39;-Dkarate.options=--tags @happyPath classpath:services/getirFood/AbortBasket&#39; -f pom.xml</pre><p>So, we need a suite name and test case name to run a single test.</p><h4><strong>1. Services</strong></h4><ul><li><em>Saving the prepared scenario pairs for maven run.</em></li></ul><pre>@GetMapping(&quot;/karate/{process}/{runId}&quot;)<br>public KarateScenarioResponse setKarateScenario(@PathVariable String process, @PathVariable String runId) {<br>    Karate_scenarios scenario = new Karate_scenarios();<br>    Date now = DateUtils.<em>addHours</em>(new Date(), 3);<br>    scenario.setScenario(process);<br>    scenario.setStatus(&quot;ready&quot;);<br>    scenario.setRunId(runId);<br>    scenario.setCreatedDate(now);<br>    if (karateScenariosRepository.findOneByScenario(process) == null) {<br>        karateScenariosRepository.save(scenario);<br>        return new KarateScenarioResponse(scenario);<br>    } else {<br>        return new KarateScenarioResponse(karateScenariosRepository.findOneByScenario(process).get_id(), runId, process, &quot;Test already defined&quot;, now);<br>    }<br>}</pre><ul><li><em>Getting a single available scenario</em></li></ul><pre>@GetMapping(&quot;/karate/available-process&quot;)<br>public KarateScenarioResponse getAvailableKarateScenario() {<br>    lock.lock();<br>    try {<br>        Karate_scenarios karate_scenario = karateScenariosRepository.findOneByStatus(&quot;ready&quot;).get(0);<br>        if (!karate_scenario.equals(null)) {<br>            Date now = DateUtils.<em>addHours</em>(new Date(), 3);<br>            karate_scenario.setStatus(&quot;in progress&quot;);<br>            karate_scenario.setLastTestedDate(now);<br>            karateScenariosRepository.save(karate_scenario);<br>            lock.unlock();<br>            return new KarateScenarioResponse(karate_scenario);<br>        } else {<br>            lock.unlock();<br>            return new KarateScenarioResponse();<br>        }<br>    } catch (Exception e) {<br>        lock.unlock();<br>        return new KarateScenarioResponse();<br>    }<br>}</pre><ul><li><em>Updating the scenario status</em></li></ul><pre>@GetMapping(&quot;/karate/{id}/status/update&quot;)<br>public KarateScenarioResponse updateKarateScenarioStatus(@PathVariable String id) {<br>    Karate_scenarios karate_scenario = karateScenariosRepository.findOneBy_id(id);<br>    if (!karate_scenario.equals(null)) {<br>        karate_scenario.setStatus(&quot;done&quot;);<br>        karateScenariosRepository.save(karate_scenario);<br>        return new KarateScenarioResponse(karate_scenario);<br>    } else {<br>        return new KarateScenarioResponse();<br>    }<br>}</pre><ul><li><em>The database should look like the below. As you can see, each task has a status. Tasks are in “ready” status when the scenario is prepared by the Jenkins job and ready to run. While they are running in containers, tasks switch to “in progress” status and after the test is finished, they go to “done”. The “runId” field is a task run identifier from Jenkins and it’s used in the folder the test reports in the end.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHrL7K7PklBfmkjes30k3g.png" /><figcaption>Scenario records on MongoDB</figcaption></figure><h4>2. A cron job that prepares the scenarios via Jenkins</h4><ul><li><em>PrepareKarateScenarios.groovy</em></li></ul><pre>def features = []<br><br>def findTags(feature) {<br>    def folder = &quot;$<strong>{</strong>env.WORKSPACE<strong>}</strong>/src/test/java/services/getirFood&quot;<br>    def file = new File(&quot;$<strong>{</strong>folder<strong>}</strong>/$<strong>{</strong>feature<strong>}</strong>&quot;)<br>    def lines = file as String[]<br>    def tags = lines.findAll <strong>{ </strong>it.trim().startsWith(&#39;@&#39;) <strong>}</strong>.collect <strong>{<br>        </strong>it.replaceAll(&quot;\\s&quot;, &quot;&quot;)<br>    <strong>}<br>    </strong>for (int z = 0; z &lt; tags.size(); z++) {<br>        String url = &quot;https://bft.getir.test.com/prepare-scenario/karate/$<strong>{</strong>feature<strong>}</strong>,$<strong>{</strong>tags[z]<strong>}</strong>/$<strong>{</strong>env.BUILD_NUMBER<strong>}</strong>&quot;<br>        def sendScenarioToBft = new URL(url).getText()<br>        println(sendScenarioToBft);<br>    }<br><br>    return tags<br>}<br><br>pipeline <strong>{<br><br>    </strong>agent <strong>{ </strong>label &#39;java-karate&#39; <strong>}<br><br>    </strong>stages <strong>{<br>        </strong>stage(&#39;Prepare Scenarios&#39;) <strong>{<br>            </strong>steps <strong>{<br>                </strong>script <strong>{<br>                    </strong>final foundFiles = findFiles(glob: &quot;src/test/java/services/getirFood/*.feature&quot;)<br><br>                    for (int i = 0; i &lt; foundFiles.length; i++) {<br>                        def filename = foundFiles[i].name<br>                        features &lt;&lt; filename<br>                    }<br>                    for (int y = 0; y &lt; features.size(); y++) {<br>                        def process = findTags(features[y])<br>                    }<br><br>                <strong>}<br>            }<br>        }<br>    }<br>}</strong></pre><p>This cron pulls the api-automation-test repo from git and sends the $<strong>{</strong>feature<strong>}</strong>,$<strong>{</strong>tags<strong>}</strong> info to the database <strong>as a single scenario </strong>that can run after analyzing the .feature karate files.</p><h4>3. Runner script</h4><ul><li><em>runner.py</em></li></ul><pre>import requests<br>import json<br>import subprocess<br>import os<br><br><br>def get_scenario():<br>    url = &quot;https://bft.getir.test.com/prepare-scenario/karate/available-process&quot;<br>    return json.loads(requests.request(&quot;GET&quot;, url).text.encode(&#39;utf8&#39;))<br><br><br>def prepare_command(tag, feature):<br>    command = &quot;mvn clean test -Dtest=TestRunner -DfailIfNoTests=false -DstoppingStep=null &#39;-Dkarate.options=--tags &quot; + tag + &quot; classpath:services/getirFood/clientApiGateway/&quot; + feature + &quot;&#39; -f pom.xml&quot;<br>    return command<br><br><br>def run_cmd(command):<br>    return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()<br><br><br>def sync_project():<br>    run_cmd(&quot;rm -rf api-automation-test&quot;)<br>    clone = &quot;git clone --shared --branch feature/QA-739 https://xxx:xxx@bitbucket.org/xxx/api-automation-test.git&quot;<br>    run_cmd(clone)<br>    os.chdir(&quot;api-automation-test&quot;)<br><br><br>sync_project()<br><br><br>def sendReport(runId):<br>    fileName = run_cmd(&quot;ls -R target/reports | grep &#39;.*\.json&#39;&quot;)<br>    pwd = run_cmd(&quot;pwd&quot;)<br>    files = {<br>        (&#39;file&#39;,<br>         (fileName.rstrip(),<br>          open(pwd.rstrip() + &#39;/target/reports/&#39; + fileName.rstrip(), mode=&#39;r&#39;),<br>          &#39;application/json&#39;))}<br>    url = &quot;https://bft.getir.test.com/report/upload&quot;<br>    headers = {<br>        &#39;directory&#39;: runId<br>    }<br><br>    requests.request(&quot;POST&quot;, url, headers=headers, data={}, files=files)<br><br><br>while True:<br>    response = get_scenario()<br>    if response[&#39;status&#39;] == &quot;in progress&quot;:<br>        tag = response[&#39;scenario&#39;].split(&quot;,&quot;)[1]<br>        feature = response[&#39;scenario&#39;].split(&quot;,&quot;)[0]<br>        command = prepare_command(tag, feature)<br>        run_cmd(command)<br>        sendReport(response[&#39;runId&#39;])</pre><p>This python script pulls the repo and checks if there is an available single Karate scenario waiting to run and runs it with maven.<strong> Thus, if we put this script into a container, it will search for a single test to run until the container is stopped. </strong>We need a container that has JDK and executes python. Let’s have a look.</p><h4>4. Dockerize</h4><ul><li>Dockerfile</li></ul><pre>FROM maven:3-jdk-8-alpine<br><br>ENV <em>PYTHONUNBUFFERED</em>=1<br><br>RUN echo &quot;**** install Python ****&quot; &amp;&amp; \<br>    apk add --no-cache python &amp;&amp; \<br>    if [ ! -e /usr/bin/python ]; then ln -sf python /usr/bin/python ; fi &amp;&amp; \<br>    \<br>    echo &quot;**** install pip ****&quot; &amp;&amp; \<br>    python -m ensurepip &amp;&amp; \<br>    rm -r /usr/lib/python*/ensurepip &amp;&amp; \<br>    pip install --no-cache --upgrade pip setuptools wheel &amp;&amp; \<br>    if [ ! -e /usr/bin/pip ]; then ln -s pip /usr/bin/pip ; fi<br><br>RUN pip install requests<br>RUN apk update &amp;&amp; \<br>    apk upgrade &amp;&amp; \<br>    apk add git<br><br>RUN mkdir -p /usr/src/app<br>WORKDIR /usr/src/app<br><br>RUN echo &quot;PWD is: $<em>PWD</em>&quot;<br><br>COPY Runner.py Runner.py<br><br>ENTRYPOINT [&quot;python&quot;, &quot;Runner.py&quot;]</pre><p>We containerized our script which is okay, but where should the container be running? If we run this container, it will take an available test and run it and repeat the process for each test. It’s working but this is inefficient as you have to wait a long time if you have more than 3,000 test cases. We needed multiple containers so we could <strong>run test cases in parallel. </strong>We found the <a href="https://www.portainer.io/">Portainer</a> to be the best solution for us in means of scaling and orchestrating containers and it has an easy-to-use GUI.</p><h4>5. Docker Scaling</h4><ul><li>Deploying our stack to the Portainer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O1o29reNpH4LrJntyYa3VQ.png" /><figcaption>Creating a new stack on the Portainer</figcaption></figure><ul><li>We are now able to scale our container with the <strong>docker swarm</strong>. 500 containers ask for a single test and after running the test and in the end, they will send the report to S3.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PxFGyuowfPWnDKJxqTnQWw.png" /><figcaption>Scaling containers from Services</figcaption></figure><h4>6. Reporting script</h4><ul><li>Reporter.py</li></ul><pre>import logging<br>import subprocess<br>import os<br>from boto.s3.connection import S3Connection<br><br>conn = S3Connection(&#39;xxx&#39;, &#39;xxx&#39;)<br>bucket = conn.get_bucket(&#39;xxx&#39;)<br>directory = &#39;103/&#39;<br><br>def prepare_report():<br>    os.chdir(&quot;..&quot;)<br>    command = &quot;mvn verify -DskipTests&quot;<br>    return run_cmd(command)<br><br>def run_cmd(command):<br>    return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read()<br><br><br>def download_jsons():<br>    os.chdir(&quot;results&quot;)<br>    for key in bucket.list(prefix=directory):<br>        try:<br>            res = key.get_contents_to_filename(key.name.replace(directory, &#39;&#39;))<br>        except:<br>            logging.info(key.name + &quot;:&quot; + &quot;FAILED&quot;)<br>    prepare_report()<br><br>download_jsons()</pre><p>This script downloads the JSON from the S3 bucket which we have pushed from testing in containers and triggers the maven report plugin. You are good to go with your HTML report!</p><p>It‘s clear that the team has put in extra effort and set up this flow to execute existing test packages for only a short time and as a temporary solution. Focus on how we can fix the problem rapidly so we still meet the schedule. This is how we were able to deliver our tests in time. In the meantime, we have started a new project with Rest Assured. Here is an article about <a href="https://medium.com/getir/how-to-start-a-rest-assured-project-d599181ca855">how we started a project with Rest Assured</a> written by my dear teammate <a href="https://medium.com/@fsebilet">Sebile</a>. More is on the way. Stay tuned…</p><p>Piece of advice and lesson learned: don’t choose the framework that hasn’t been released even as v1.0 :)</p><p><strong>Thanks for reading and feel free to reach out :)</strong></p><p><strong><em>Cheers</em></strong>🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc5ad83fcd1b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/getir/java-lang-outofmemoryerror-java-heap-space-in-karate-framework-dc5ad83fcd1b">“java.lang.OutOfMemoryError: Java heap space” in Karate Framework</a> was originally published in <a href="https://medium.com/getir">Getir</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load and performance test in Getir]]></title>
            <link>https://medium.com/getir/load-and-performance-test-in-getir-3bc5bf57b1e2?source=rss-5992422ece30------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bc5bf57b1e2</guid>
            <category><![CDATA[performance-test]]></category>
            <category><![CDATA[locust]]></category>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[jmeter]]></category>
            <dc:creator><![CDATA[Gökhan KARAMAN]]></dc:creator>
            <pubDate>Wed, 19 Jan 2022 16:48:47 GMT</pubDate>
            <atom:updated>2022-01-24T07:11:50.930Z</atom:updated>
            <content:encoded><![CDATA[<p>Touching millions of users requires a lot of care. Your services should be ready for users in every scenario. As Getir testers, we have been dealing with huge amounts of load on our services, so we needed to try a number of load testing tools and frameworks. We created various load test scenarios and cases to understand which is the best tool for us and see if the requirements match our limits.</p><ul><li>Jmeter</li><li>Locust</li><li>Gatling</li><li>Webload</li></ul><p>Considering our needs, the preference for open source and taking scalability into account, we have chosen Locust. Let’s explore the details of our selection and why most of the testers choose Locust.</p><figure><img alt="https://www.sozcu.com.tr/2021/ekonomi/istanbul-bogazindan-gecen-gemi-sayisi-2020de-de-azaldi-6281244/" src="https://cdn-images-1.medium.com/max/1000/0*vt5d_CfUEAex7hML.jpg" /></figure><h4>Why did we choose Locust?</h4><p>Jmeter is based on JVM so it makes things slower. Locust is an open source framework that is developed in Python. In addition to the <strong>test as a code</strong> feature, it’s highly scalable as it’s a purely event-driven application. For this reason, there is a large audience that prefers Locust over JMeter. We have determined some important features of the tools we used.</p><p>Check it out:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2fgyPvCHBdZduxMj-Yqdw.png" /></figure><p>We realized that if we run the same script with the same scenario and same duration, <strong><em>Locust is able to make more requests and the CPU usage is almost half as much</em></strong>. If we want to reach the same RPS with the Jmeter, we need to use more CPU, so our cost increased a lot when it was desired to run these tests in the Cloud.</p><p><strong>Jmeter</strong> has a structure that assigns one thread for each user activity. If we try to run the script on a standard machine and we observed that its RPS amount does not meet Getir’s needs. Especially in GUI mode, it gets an “OutOfMemoryError” error. It’s a bit more effective in non-GUI mode and we can probably solve this memory error by optimizing java heap size, but is this really necessary? Particularly when we already have a better tool that is well optimized. Yes, <strong>Locust </strong>has a different perspective about user simulation which uses an asynchronous thread approach. It allows to easily simulate simultaneous users on a single machine while performing complex tests with many steps in it. <strong>We have observed that this approach allows us to ensure more users with the same conditions in our tests.</strong></p><h4>So, how do we do that?</h4><ul><li>First of all, we have a <strong>kick-off</strong> meeting to determine the requirements with the whole team. We discuss the user scenario and measure the weights of each endpoint according to the old data. By the end, we know how much load we need to supply for each service.</li><li>Then we <strong>develop the test script </strong>and make a cold shot.</li><li>We are <strong>deploying</strong> our load test script to an EC2 service that has auto-scale workers to the needs of RPS amount (Locust has already supplied auto-scale).</li><li>We<strong> prepare a pre-prod env</strong> that is similar to the production environment, ensuring that it’s the same data and same env with the same numbers of pods.</li><li>Then, we<strong> run</strong> the test script.</li><li>We <strong>monitor</strong> the services to see where our bottlenecks are.</li><li>Finally, we <strong>report </strong>back<strong> </strong>to the team.</li></ul><p><strong>Thanks for reading and feel free to reach out :)</strong></p><p><strong><em>Cheers</em></strong>🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bc5bf57b1e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/getir/load-and-performance-test-in-getir-3bc5bf57b1e2">Load and performance test in Getir</a> was originally published in <a href="https://medium.com/getir">Getir</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>