As a software quality and testing professional (SDET/ QA Engineer / QA Specialist/ Test Automation Engineer etc ..) you probably got this question a few times “What is our current testing coverage?”. The expectation by the person posing the question would typically be an answer containing a percentage value, with the benefit of confirming to them that a value close to or equal to 100%, the product is “thoroughly” tested, ready to deploy, and bug-free.

Well, whatever the value is, it does not necessarily give these indications. It also depends on to who this question is asked. A backend/frontend developer will think about the unit test coverage of their services and quickly get that value from the latest CI pipeline job run which would report a percentage line coverage with a tool such as Jacoco and Istanbul.

A software tester, on the other hand, would think about the test coverage from a different perspective. They might consider the different types of tests that have been run, such as integration tests, system tests, and acceptance tests. They might also look at the E2E tests of the business user scenarios that have been covered, and whether those scenarios are sufficient to ensure the quality of the product, keeping in mind how the product will be utilised and which risks might have the highest impact. They would consider the functional test coverage of their latest test plan execution, and will report a percentage value with the result from an HTML report of a tool like Selenium, Cypress or Playwright.

In both cases, the focus is on test coverage, but the definition of coverage and the methods used to measure it can vary greatly. As we go further up the imaginary pyramid of Testing, the measurement as well as the contents of the measurement gets more abstract and more opinionated. However, at the lowest level, ie Unit Tests, things are more clear.

Both views provide important metrics that help in tracking the progress and effectiveness of the testing efforts during the development cycle. However, they do not give an indication of the quality or strength of the tests themselves. This is where Mutation Testing comes into the picture.

Unit test coverage != Unit test strength

Mutation testing is a technique used to evaluate the quality of a set of unit tests by introducing small, intentional changes, known as mutations, in the code under test. These mutations are designed to simulate the types of bugs or defects that the unit tests are expected to find. The idea is that if a set of unit tests is able to detect and fail the mutated code, it means that the tests are strong enough to find real defects.

For example, consider the following Java code:

package com.mutant.example

public class IntegerIdentification {
   
  boolean isPositive(final int num) {
    return num >= 0;
  }
}

Suppose we have a unit test that calls this method and checks if the result is as expected:

public class IntegerIdentificationTest {


  @Test
  public void isPositiveTrue() {
      IntegerIdentification integerIdentification = new IntegerIdentification();
      Assert.assertTrue("Integer was not positive", integerIdentification.isPositive(2));
  }
  
  @Test
  public void isPositiveFalse() {
      IntegerIdentification integerIdentification = new IntegerIdentification();
      Assert.assertFalse("Integer was positive", integerIdentification.isPositive(-1));
  }

}

A mutation testing tool like Pitest would create a mutation in the isPositive method by replacing the ≥ operator with a > operator, resulting in the following code:

boolean isPositive(final int num) {
    return num > 0;
}

If the original unit test suite is able to detect this change and fail the test, it means that the tests are strong enough to detect real defects (or unintentional changes) in the code. On the other hand, if the tests pass, it indicates that the they are weak and there is conditions which are not covered by the tests. Running Pitest in this case will therefore report that the existing tests are weak as a change in the implementation was not covered by a test case.

Mutation testing helps in identifying the weaknesses in the unit tests. This, in turn, increases the confidence in the quality of the product and reduces the risk of defects being introduced in the final product. Pitest tries to ‘attack’ the code with mutants and then report a Mutation Coverage and Test Strength percentage alongside the traditional Line Coverage percentage.

Tools are helpful as long as the team is onboard

Introducing a tool can improve the team’s output as long as the team is onboard with adapting it, with the right mindset. Pitest allows you to define a mutation coverage and/or test strength threshold so that the codebase does not degrade below a defined percentage. This could be used as another quality gate within CI/CD so that the build fails when conditions are not met. Having the team onboard means that the corrective measures are taken and unit tests are improved as much as possible. Otherwise, the additonal check will be quickly turned off the next time a build fails and something needs “to be quickly deployed”. So make sure that the team understands the concept and the benefits before adopting Mutation Testing! Let’s defeat those mutants!

Weak Unit Tests? Mutants to the rescue! was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Onto Part 2

In my last post in which I introduced the Timings API and briefly described the ELK stack, we concluded at just before applying Timings to our previously created functional tests. In this part, we will modify those tests to start asserting them against some necessary performance checks!

First off, we need to understand the different Timings functions that are available for us to use. One of these is the’/apitiming’, which we can use to assert the performance of API calls. We will not get into this endpoint since the focus of our post is further up the stack, on the functional browser tests. So let’s move on to explaining the two primary endpoints which we will use in our tests: ‘/navtiming’ and ‘/usertiming’

Applying it to our tests

We will use the same basic test framework and test scenarios that have been used for our previous WebdriverIO posts, and extend on these test scenarios as necessary. We will be using this repo for the code examples in this post. The first action we have to take is to switch to the timings-docker repo and run the docker-compose setup that we discussed in Part 1. To make sure everything is up, open three browser tabs and navigate to each respective service that should be up, i.e.

• Timings (ex: http://localhost )
• Elasticsearch (ex: http://localhost:9200)
• Kibana (ex: http://localhost:5601)

Obviously, these URLs depend on where you are hosting your setup, which you can easily configure through a config file. You can find more info about creating your config file from the timings-docker Github repo.

The next step would be that of installing and adding the timings-client npm package to our package.json so that its added as a dependency. To do this just run this command within the root of our WebdriverIO tests repo:

npm install --save-dev timings-client-js

We will modify and start monitoring our tests within /test/WebdriverIO_site/navigationalLinksTest.js, so to do this we need to add these two constants to be able to use the API:

const timings = require('timings-client-js'); const perf = new timings.PUtils('timings.conf.js');

Now that we have all the necessary setup, we can proceed to modify each test scenario. In each ‘it’ scenario, we can actually overwrite API parameters such as the expected page load time for that particular test. To do this, add the following line at the beginning of each test:

const perfParams = perf.getApiParams({});

To overwrite the default settings, you can use it such as:

const perfParams = perf.getApiParams( { "sla": {"visualCompleteTime": 2000} } )

Async/Await

The first significant change that we need to do to our existing tests is to convert them to use NodeJS async/await functionality to enforce the test’s synchronous flow. Even though WebdriverIO is run synchronously through the use of the wdio-sync package and enforced with the parameter ‘sync:true’ within wdio.conf.js, some code execution can still have issues. Therefore to prevent that, change the declaration of the function as follows:

it('should go to Developer Guide page when choosing Developer Guide link', async function () { ...

Now this will enable us to use the await call in front of methods which we need to execute in synchronous mode.

Navtiming vs UserTiming

To proceed with modifying the tests, we need to understand how each scenario is interacting with the site. This will help us know which Timings API call to execute.

The first scenario is “it(‘should go to the Developer Guide page when choosing Developer Guide link’)”. This scenario checks that when a user lands on the Webdriver.io homepage and clicks on the Developer Guide menu item, arrives on the Developer Guide page. If we inspect the page to open the Chrome Dev Tools and go to the Network tab, we can check if the link causes a full page load or essentially loads data already stored with the current page. In our case, all links from the top navigational bar cause a full page load. As instructed in the Timings documentation, ‘/navtiming’ should be used for full page loads. So to use this call, we will add the following code:

const injectJs = await perf.getInjectJS('navtiming', 'visual_complete', true); const injectCode = injectJs.data.inject_code;

To get the performance data from the browser we need to store the response to our browser actions. Therefore, we need to add a constant and chain the actions performed on the browser object such as:

const injectCodeResponse = await browser .url(https://medium.com/'http://webdriver.io/') .waitForVisible('[alt="WebdriverIO"]') .click('=Developer Guide') .isVisible('[id="Developer-Guide"]') .execute('window.performance.mark("visual_complete");') .execute(decodeURIComponent(injectCode));

We can now continue with the actual functional assertion which we previously defined in our test:

assert.equal(await browser.getUrl(), 'http://webdriver.io/guide.html');

Perfect! Now we can grab the browser response as this has the performance data:

const injectCodeResponseValue = injectCodeResponse.value;

To send the performance data to the API, we need to call the endpoint with the following command:

const navtimingResponse = await perf.navtiming(injectCodeResponseValue, perfParams);

Now that we also have the return object from the ‘ perf.navtiming’ call, we can check the assertion against our SLA configuration:

if (navtimingResponse.data) { const apiResponse = navtimingResponse.data; expect(apiResponse.assert, 'Performance failed! assert field is False \nNavtiming: '+JSON.stringify(apiResponse.export.perf, null, 2)).to.be.true; } else { console.error('API error: ' + JSON.stringify(navtimingResponse, null, 2)); }

That’s it! This way we can make sure that our test not only asserts its functional assertion but also the performance threshold.

So where should we use /UserTiming?

To show an example of ‘/usertiming’, I will create a new test against the site https://reactjs.org/. Therefore, I will now structure the tests folder so that there is a folder WebdriverIO_site and another one called ReactJS_site so that we can distinguish between each target site for our tests. For WebdriverIO to be able to execute these tests within a different path, we need to go to the wdio.conf.js file and change the pattern defined under specs to ‘./test/*/*.js ‘

Great! Now we can create the new test. This test will consist of another simple navigational test. The user should go to the Getting Started page when he clicks on the ‘Docs’ link on top. The only few differences in the implementation are easily detected. First off, the ‘ perf.getInjectJS’ call will now have ‘ usertiming ‘ as its first parameter value while the second parameter can now be set to an empty string. Since the page won’t reload with the action performed by the user, we will set a ‘start’ and ‘stop’ mark to measure the duration of the test. You can have a look below at the full implementation of the test:

const assert = require('assert');
const timings = require('timings-client-js');
const perf = new timings.PUtils('timings.conf.js');

describe('ReactJS Navigation Links', async function () {
    it('should go to the Getting Started page when choosing Docs link', async function () {

        const perfParams = perf.getApiParams({sla: {pageLoadTime: 5000}}); //you can overwrite values in perf.js
        const injectJs = await perf.getInjectJS('usertiming', '', true); //Request inject code from API - `true` = strip querystring
        const injectCode = injectJs.data.inject_code;
        const injectCodeResponse = await browser
            .url(https://medium.com/'https://reactjs.org/')
            .execute('performance.mark("demo_start");') // Set User Timing "start" mark
            .click('=Docs')
            .isVisible('Getting Started')
            .execute('performance.mark("demo_stop");') // Set User Timing "stop" mark
            .execute(decodeURIComponent(injectCode)); // Inject JS code into browser object

        assert.equal(await browser.getUrl(), 'https://reactjs.org/docs/getting-started.html');

        const injectCodeResponseValue = injectCodeResponse.value; // Grab the browser's response - has the perf data!
        const usertimingResponse = await perf.usertiming(injectCodeResponseValue, perfParams); // Send perf data to API

        if (usertimingResponse.data) {
            const apiResponse = usertimingResponse.data; // Grab the API's response - has the assert field!
            expect(apiResponse.assert, 'Performance failed! assert field is False \nNavtiming: ' + JSON.stringify(apiResponse.export.perf, null, 2)).to.be.true; // Assert the result!
        } else {
            console.error('API error: ' + JSON.stringify(usertimingResponse, null, 2));
        }

    });


});

The Result

Now we can run both specs by running npm run test in the terminal. If we have set the configuration and tests correctly, you will see five passed tests from two specs. As for further information on the performance results, the Timings Dashboard found in your Kibana URL will display details on the outcome of all the test runs. Under the Visualize menu in Kibana, you will find loads of helpful visualizations such as SLA vs Actual and NavTiming metrics line charts which will show trends in your tests and progress over time. These metrics can be highly significant when we need to detect slowness with upcoming versions of the application. A shift-left in performance testing will prevent the deployment of a newer version of the app with performance issues.

Originally published at https://testautonation.com on 08/2018.

Time those functional tests with Timings API — Part 2 was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In my earlier post which discussed the topic of who owns quality within cross-functional agile teams, I mentioned that there are various factors that we should consider for testing. Performance is a non-functional aspect which testers rarely have time to cover. It can already be quite challenging to satisfy a substantial amount of functional testing with both manual and automated efforts. What if I tell you that you can add a layer of performance checks with minimal effort? I present to you, Timings.

Last October, I had the opportunity to attend the SeleniumConf testing conference in Berlin. One of the sessions was about the inclusion of performance assertions to standard functional testing. In this session, Marcel Verkerk introduced Timings, with the concept of adding performance monitoring to existing functional test suites. These tests can be functional end-to-end browser automation as well as API tests. In this post, I will explain how to locally set up the necessary tools to be able to make use of the Timings library and visualise the performance results of your tests using ELK.

What is Timings?

Timings is an open source npm module which provides you with an API which offers the functionality for communicating with the W3C Performance API available in all major browsers. This API within the browser collects timestamps for all the events within the browser, allowing you to calculate metrics such as the page load time and the Time to First Byte. You did not know about it? Go to your browser’s console and type performance.timing. With this communication between the Timings API and the W3C Performance API, you can then measure and assert the performance of your tests.

ELK who?

The Timings API is capable of transferring results to the ELK stack. ELK stands for ElasticSearch, Logstash, and Kibana. These three open-source tools allow you to transform, store and visualise logs without spending a dime on proprietary tools. This is essential so that we can view and compare the values from multiple test runs, especially with different system builds.

In simple terms, this Application Logs Management tool can concatenate multiple sources of logs, transform them for storing in the same database, and then visualise. There is usually no consistency between different generated logs. Therefore issues such as different time formats make it hard to amalgamate separate logs together. We can say that this process is done by Logstash and Elasticsearch. Another important fact is that logs can contain all sorts of information, and not just server uptime or response times. We can get information such as how many requests were done per day, or how many purchases were completed this month. However, logs are not usually available to the persons for whom they would be most valuable. The ELK stack helps in transforming this information to visualise it through Kibana dashboards efficiently.

Let’s get it running!

Marcel Verkerk has made it very easy for us to set up a local instance of the ELK stack. The primary requirement for this would be the installation of Docker, for which you can also find installation requirements and description within Marcel’s repo readme (https://github.com/Verkurkie/timings-docker). Once you have cloned the repo and created a custom config based on your needs, you are now ready to go! Run docker-compose up within the repo, and this will start loading up Elasticsearch, Kibana and the Timings API. Once up, you can check if they are up and running by checking the below links for each container.

• Timings — http://your_server
• Elasticsearch — http://your_server:9200
• Kibana — http://your_server:5601

So, how to use this ‘Timings’ anyway?

The idea is that you can add a layer of performance measurements with an existing test suite of functional verification. This allows testers to safeguard the performance of a system by blocking builds that would degrade the user experience.

As you can expect, we do not achieve the goal just by getting the performance data. We have to set up monitoring dashboards and automated notifications in order not to ignore this additional information. Therefore, when planning to gather performance results, make sure also to plan who will be responsible for it.

To first gather this data, we need to add some calls to our existing browser tests. With every page load or user action within the test, we need to inject some code allowing us to then retrieve the performance metrics. Then, depending if the measured interaction results in a full page load or a soft page load, different calls (navtiming or usertiming API call) need to be used. If tests are setup correctly, you can end up gathering some useful data which out of the box will show up on a dashboard similar to the one shown above.

…to be continued

That’s it for part 1 of my ‘Timings’ post, in which I went through a high-level explanation of how one can use this library to make use of the browser’s performance API with an ELK stack. In part 2 I will go into detail of how we can implement the Timings calls within the WebdriverIO tests created in my previous .

Originally published at https://testautonation.com on 05/2018.

Time those functional tests with Timings API — Part 1 was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a previous post, we showed how we could track the duration of our existing automated functional tests, allowing us to detect any anomalies between different builds. Timings API gave us the opportunity to use the same tests for both functional verification as well as performance. However, this may not be the best way to be sure that under specific (regular or higher than normal) traffic conditions, the site or application still performs as expected. To ensure that the system behaves normally under load, we need to generate simulated traffic and measure performance-related metrics. One of the solutions that are growing in popularity is Gatling.

Gatling offers both an enterprise solution called Gatling Frontline as well as a free open-source tool. In this post, we will mention only the free open-source version of Gatling. One of the highest selling points of this tool is the way its build on top Akka allowing the simulation of thousands of virtual users from a single machine. On the contrary, JMeter would need more resources to create the same amount of simulations. Not to forget that it has its own expressive DSL allowing code-based scenarios to be maintainable and stored within version control systems.

Types of Performance Tests

When we talk about Performance Testing, there are different reasons and situations why we would want to run these checks. For example, we would want to know the upper limits of capacity that the system can handle before it degrades. We can also run a specified load for a number of hours to make sure that performance keeps at an acceptable level for a long time. Another scenario would be that of ensuring that a new system build persists the same quality of performance as the current live build.

With Gatling, we can simulate a user scenario for a frontend website by using the provided Recorder tool. This tool allows you to set up a proxy and connect your browser to it, enabling you to then go once through the scenario as a typical user would and be able to automatically generate a script. This script will be then used to repeat the scenario for the desired amount of virtual users.

Getting started using Maven

To quickly create scenarios, you can start a new Maven project by using the Maven Archetype: -highcharts-maven-archetype. To do this, follow the screenshots below. The example is shown using IntelliJ IDE.

Once ready, Enable Auto-Import so that Maven downloads all the required dependencies. You should be able to see a similar project structure as shown below.

Let’s create a Scenario

Now the simplest way to start creating your scenarios is to use the Gatling Recorder. Right click on the Recorder file under the ‘ scala’ folder, and choose ‘Run Recorder’. This will launch the Recorder window.

There are two ways you can make use of the Recorder to generate your simulation, the first one is to set up the proxy and connect your browser or OS to it, or else you can save a HAR file from your browser. In this post we will go with the latter.

The scenario that we will generate is very simple. We will open the site http://testautonation.com, and then go to the TestAutonation Team page. To create our HAR file, let’s open a new Chrome window and open the Chrome Dev Tools by right-clicking on the page and choosing ‘Inspect Element’. Now we can perform the required actions to land on the mentioned page. Once done, go to the Network tab in the Dev Tools, right-click on any item and choose ‘Save as Har with Content’.

Now in the Recorder, chose the HAR file and give a meaningful class name to the simulation. The rest of the options left unchanged. Go on and press the ‘Start’ button. The Recorder will prompt that it successfully converted the HAR file to a Gatling simulation, so we can close the Recorder window.

Next thing to do is set up our scenario. If you open the newly created simulation class and go to the very end, you will find the ‘setUp’ call which is currently set to run the simulation for 1 ‘atOnceUsers’. We can modify this to create our required simulated traffic. Gatling provides different ways of setting up your scenario which can be found on this page. For this example, we will try out the ‘ rampUsers’ during a defined duration and we will generate 100 virtual users during 1 minute. We can do this by setting the ‘SetUp’ call to this:

setUp(scn.inject(rampUsers(100) during (1 minute))).protocols(httpProtocol)

Once the simulation is defined, we can go ahead and run the Engine class. This will prompt you to optionally enter a description for the simulation run which will be then shown in the generated report. At the end of the simulation run, you will be shown the path for the report which you can open in a browser window. This will provide you with loads of information on how your site performed against the simulated traffic.

What’s the Trend?

Once you decide which scenarios should be re-run for each new build of the site, a great way to track the trend of test results is to run your performance tests using a CI tool such as Jenkins. This way you can easily check previous runs and assess if a new build is degrading or improving the site. To make your life even easier, you can make use of a plugin which plots the results from the available test run to show graphs such as the average response time. The plugin can be found here.

That’s it? What can be improved?

If you had a quick look at the generated code you will notice that it is currently not so easy to read and maintain. It definitely needs a refactoring session. We will discuss this and more on the usage of Gatling in the next post. Until then, let us know if you have any comment or share your insight on Gatling with us!

Originally published at https://testautonation.com on 01/2019.

Using Gatling to track your site’s performance was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s digital world, taking security with a light touch is risky, to say the least. Despite the apparent validity of this statement, some industries appear to invest lightly on security within their software development.

From the perspective of Software Development professionals, one would think that the company would ensure that security tasks are tackled within the development teams. Security-related enhancements would pop up within sprint commitments and Agile task boards. It is a fact that various ways and measures can be taken to protect a web application. As an example, networking and infrastructure can have security at the heart of each decision, protecting the system from unauthorized access through comprehensive whitelisting and/or blacklisting. But the application has to be built with the same characteristic in mind because if discovered, one security hole can make the whole system at risk.

Is it part of our job?

Let’s cut to the chase. Does security fall under the responsibilities of QA Engineers? The answer can be debatable and like most answers, it depends. Some workplaces might define it explicitly under the QA’s tasks, while others might not even mention it at all. There are also specific roles which are assigned with this responsibility, such as Security Engineers and IT Security Administrators. These roles not only ensure that systems do not expose any vulnerabilities but also have to prepare for the effect of a natural disaster by developing disaster recovery plans.

Having said that, as a QA, quality is in our heart, and security affects quality as it influences the user experience directly. Since security is another non-functional aspect of an application, its effects do not stand out as much as the user’s direct interaction with the system’s functionality. We need to dig deeper and truly understand the internals of a system under test. However, to be able to do this, testers need to broaden their knowledge and learn about security vulnerabilities.

Where do we begin?

Like every other topic, there is loads of information on the internet. One of the best sources for Security Testing is OWASP. This community is an excellent source of open-source security tools. From blog posts to audio podcasts, to handy open-source tools, they have it all. Another great source which I highly recommend is the Secure Guild conference organized by Joe Colantonio.

There are a lot of relevant tools out there. Paid and free tools with the ability to hook to IDEs and report issues, or scan your code and detect problems. The OWASP ZAP is a dynamic tool which tries to identify security vulnerabilities using a local HTTP proxy. It can actively or passively attack your system and report vulnerabilities. ZAP is part of the OWASP set of tools. In fact, even the Juice Shop which Andrea discussed in a previous post, is also part of OWASP tools.

In this post, I will focus on the ZAP API Scan, which is a script included with the official ZAP Docker images. The only dependency that it has is Docker itself. The API runs an active scan which is tuned to detect vulnerabilities for APIs, therefore excluding checks such as XSS (Cross-Site Scripting) which relies on scripts running within the browser. As a side note, an active scan is an actual attack on the target application, which means that you should be careful of how and where to run it.

Let’s try it out!

For the script to be able to run against your API, it needs the API documentation in the form of either OpenAPI or SOAP, defining it through the -f parameter. If your application is serving another type of API documentation such as RAML, you will need to use a converter. In the case of RAML, one of these converters can be found on this GitHub, allowing you to convert different RAML versions to OpenAPI. You can either specify a local file or point the API to the URL which contains the API specification. Do this by using the -t parameter. You also need to specify which format you are using by setting the -f parameter to either ‘openapi’ or ‘soap’. You can have a look at the complete list of available parameters in this Wiki.

A typical run would look like this:

• First, we pull the Docker image by running this in the terminal:
  docker pull owasp/zap2docker-weekly
• Then we run the actual scan:
  docker run -v /tmp:/zap/wrk -i owasp/zap2docker-weekly zap-api-scan.py -t https://www.yourapp.com/openapi.json -f openapi -r ScanReport.html

At the end of the process, you will be able to access the defined HTML report file where you will see if the scan has found potential risks. The report will not show the full list of risks that the script checks. You can see the whole list (passed and failed) from the terminal.

Alert!

The report itself gives a description of the alerts being raised, together with their risk level. In the screenshot of a sample report shown above, you can see that apart from the explanation of the alert, the report also provides a solution to fix it. Let’s have a closer look at this alert.

The ZAP report mentions that the X-Content-Type-Options header is missing from one or more responses. Setting this header to ‘nosniff’ will prevent the ability of a browser to override the content type through MIME sniffing. This sniffing is intended for beneficial use such as when the browser needs to interpret the content but does not have the necessary information to do so. In this case, it tries to understand what the content is by inspecting the response bytes and detecting the format of the file. However, this can also be used maliciously and expose the app to attacks through MIME-type confusion. Most modern and updated web browsers do not allow sniffing, however, the safer way would be to enforce it explicitly.

What next?

The ZAP automated script might be your first step into security testing. Ideally, the next thing would be to discuss the scan result with your team and figure out which items need changes. After that, you should start getting acquainted with web application security by researching about top 10 security risks, essentially getting familiar with web security technical words. From there, you’ll primarily learn about more tools and more ways; both manual and automated, of how to uncover vulnerabilities.

In the end, having initiative is the best approach to show how invaluable you are as a professional. Plus it adds skills to your testing tool belt!

Originally published at https://testautonation.com on 06/2019.

ZAP! Your way into Security Testing! was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Quest for Quality

The pursuit of quality is a never-ending battle. Testers around the world are continually fighting to let fewer bugs into production. They are also demanding better state of code within tickets before developers rush them for testing. When teams add improvements and new features to a system, it also implies that the required effort related to the overall quality is increasing. This matter appends to the complexity also mentioned in our previous post about the test pyramid, that the agile environment already imposes time pressures in reaching sprint goals. The representation of quality is found in various features of the product. It is not just within the correct functionality of the product that the user perceives quality. There is also the non-functional aspects such as performance, security, accessibility and browser/device compatibility. These also contribute or negatively impact the overall quality of software.

“Tester! Do your job!”

There is a concerning thought in development teams that testers solely own quality. Developers from time to time, tend to trust their faith blindly on their coding skills as well as the skills of testers by skipping their quick checks. It is a fact that testers are continuously trained and focused on preserving the quality of a product, however, remember that the output is the result of full team collaboration. The effort towards completing sprint tasks is usually the pursuit of 3 primary roles:

The Product Owner

With the multitude of quality factors within a software product, the various stakeholders of these areas must add their effort to the overall quality. First off, the product has its defined business rules. If the product owner does not explicitly describe requirements, how can testers confirm the expected outcomes? Therefore the product owner must add quality to the overall process of development by making sure that what the business wants is thoroughly communicated. Product owners are also in charge of signing off tickets, giving the final go-ahead after the one from testers. According to the official Scrum Guide, the Product Owner is accountable for: “Optimizing the value of the work the Development Team performs.” which in the end affects the product’s extrinsic quality (You can discover the details on extrinsic and intrinsic quality from Jim Highsmith’s Agile Project Management: Creating Innovative Products.

The Developers

Systems are continually changing with enhancements and additional functionalities introduced by the developers. For this, the developers must create and maintain unit tests to continuously validate the underlying logic of the system. It is sensibly unrealistic if testers would have the responsibility of checking these unit tests as well, especially with the typical imbalanced ratio between devs and testers within agile teams. Furthermore, code coverage indicates that the developers added quality gates with each new logic within the system, but that would not necessarily mean that all permutations and possible bugs have now been avoided. Therefore, developers should also for their benefit, help testers with their efforts and safeguard quality in any way necessary.

The Testers

It is clear that quality has to be continually on the mind of the tester. With each area of the system that the team modifies, various testing factors should be in consideration: functionality, security, accessibility, performance, device compatibility and much more. Unfortunately, it is rare for testers to cover all or even few of these factors. Much of the sprint time goes towards testing functionality. Testers make use of test automation to gain time and be able to cover more of these factors. Since quality is the primary focus and the persistent concern for this role, the drive for a better product is the main quest for each tester.

Definition of Done

A way to make sure that everyone owns quality within a team is to define clear testing goals within the definition of done. Evolving Agile teams will see a shift in their definition of done towards more rigorous checklists. Therefore, test automation should be part of this checklist. The team should spend time within grooming sessions to discuss which scenarios are possible and feasible to automate. The automation of these test cases would then allow testers not to cover previously tested scenarios repetitively. Also, it would be worth introducing a set of acceptance criteria related to non-functional requirements such as performance. Hence the team must be sure that with each new development, the system performance does not deteriorate. A collection of security and performance checks should be followed to control the impact caused by code changes.

Therefore, the whole team needs awareness of the testing efforts to be completed and thus plan their sprint commitments accordingly. When testers ask for help on the automation part of testing, this dev/QA effort would be going for the same goal of achieving the commitment. Testers, do not be shy about asking for help. If you have obstacles with your automation, make it a team problem.

What else?

Promote collaboration and consequently share your efforts and achievements. Involve developers with test automation, they will gain interest and will contribute to different ideas and perspectives! In the end, quality should be a team effort. Success or failure should always be in the name of the whole team. In conclusion, ensure that everyone owns quality as this is clearly another successful way of promoting it within the process.

References: Dan Tousignant, Who Owns Quality In Agile?

Originally published at https://testautonation.com in 02/2018.

Quality? I own it? You own it? We own it! was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

While the world is going through these surreal times with the COVID-19 pandemic, I wanted to share some thoughts on the impact of having to start a new job and switch to a remote working environment. I will share a few tips on what helped me and perhaps might help you in coping while working remotely. One of the factors that affect the success or failure of a Quality Assurance engineer’s job is something called credibility.

So what is Credibility?

Having started a new job right at the beginning of COVID-19 lock-down in Spain, I had to experience my first time working completely remotely for the first time from day one. The first aspect that impacted me was that the common face to face interactions, the body gestures, the work and non-work related chit-chat that goes on from attending a meeting to the coffee breaks, have been influential to strike a good relationship with colleagues in previous workplaces. That has all gone out of the window for now. But how does that affect my credibility as a QA? Why do I feel like I need to work harder to ‘prove’ myself, to be accepted as a beneficial addition to the team?

Credibility is made out of several factors, it can be defined as the perception of others on your attention to detail, knowledge, experience, attitude and input to the team. It is not an instant factor, it has to be built over time. Working remotely can hinder from transmitting the right message regarding these factors. Additionally, if you are experiencing a lack of self-belief, this will block you from portraying credibility even more.

Productivity helps Credibility

For a QA to reach a point of feeling productive, it is not a just a matter of taking a task and completing it. A QA needs to first lay the groundwork. This usually means obtaining a solid understanding of the business domain, about the product that needs to be tested. It also means asking heaps of questions and acquiring a deeper understanding of the team’s skills and dynamics. There is also the myriad of tools and processes that are used within the company, such as CI/CD. Subsequently, a QA can start identifying a reliable idea of a testing strategy.

Due to the nature of the work, preparing a strategy would be less straightforward than developing a business idea. If that wasn’t the case, I don’t think we would have bugs in every piece of software out there. As a result of the time needed for this, testers might also be slowing the team down.

It is no easy feat to feel productive while performing these tiresome but essential initial steps, and an even harder one to inspire credibility and demonstrate to your new team that you can be a valuable asset to them.

What can be helpful?

Every situation is different, and each tester might find his/her own coping mechanism. The following are just my experience and ideas of what helps to reach productivity, especially in a new job.

• Pairing with team members
  Pairing with your colleagues will effectively speed up the information flow that a tester needs to get up to scratch. A successful collaboration does not rely only on the knowledge of two persons, it needs an effective working relationship, and pairing helps in getting to know your team colleagues, especially during lock-down.
• Take one step, one task at a time.
  Starting a new role, your brain will start flooding you with possible ideas and tasks that you can start working on. But you need to control yourself and prioritize what needs to be accomplished first. What I tend to do is maintain a list of possible tasks to work on at a later stage.
• Ask, listen, communicate (and repeat)
  Don’t hold back in communicating with your peers. You might get the thought that asking questions is a sign of weakness, but it is not the case, especially when you’re the last one joining the team. Ask away, listen and grasp as much information.
• Sync up with other QAs (within or outside the team)
  If you work in a company with multiple teams assigned to different areas of a system, or completely separate projects, it would be a good idea to ping other QAs and understand how they are doing things in their team. This gives you both an insight into how they are tackling tasks as well as giving you an introduction for when you will contribute in return.
• Keep your team lead/manager, up-to-date with your progress
  At first, both you and your team lead/manager will have limited knowledge about each other’s way of working. So with a steady flow of communication, the sense of unknown will be mitigated, resulting in better assistance in case of getting stuck. It will also portray a sense of direction and an opportunity to be guided effectively. Team leads and managers are there to enable you to utilize and improve your skills and knowledge. Typically, they will anticipate that you will have a learning curve until you reach productivity. If not, make sure that you ask about what the initial expectations and goals should be.
• Non-work related sessions
  Finally, but not the least important, it is healthy and beneficial to have meeting sessions with your colleagues to just discuss non-work related topics and share the struggles of the current situation, giving an insight on how each person is coping. It might be hard at first to share, especially if you are still getting to know your teammates, but being there to listen will also help that.

Stay Positive!

The feeling of uncertainty, lack of self-belief, and loneliness as well, definitely do not help in doing an efficient job. Impostor syndrome might also kick in. However, one thought that helps to keep in mind is that there is always a solution. Communication is essential during this time. Outside of work hours, try to use your time wisely. Keep up to date with what is going on in the testing world. Have a look at this post for some good testing podcasts. Research ideas and solutions related to your new challenges at work. Try new tools and techniques and be open to different opinions and methodologies to ultimately decide using your best judgement. Stay positive!

Originally published at https://testautonation.com in 03/2020.

Tester’s Credibility — The struggles of a QA in a fully-remote world. was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

A New World

The swift and rather needed move towards Agile in the software world had led to a lot of benefits. However, it also imposed pressure on the software tester, having to deal with the accelerated turnover of development pushed forward to production. A testing strategy needs to be in place.

It is easy to figure out that repeatedly manual testing a whole system is rather unattainable. Consequently, developers and testers have to contribute effort to automated tests.

The easiest mistake that a team can make is to only rely on end to end tests. Although these tests are focused on the actual users and their relevant interaction with the system, they do not provide a quick feedback mechanism on what is wrong and where to find the underlying flaw. This is where the concept of a test pyramid comes to the rescue.

The Age of Pyramids

First brought up in the book “ Succeeding with Agile” by Mike Cohn, the Test Pyramid provides a design to help out in visualising the right balance of effort that should be allocated to the varied types of automated tests.

At the depths of the pyramid lies the foundation of good old unit tests. These provide a comforting assurance that each unit of code is operating as expected. One of the reasons behind preaching for the largest amount of tests to be at this layer is that these provide the quickest feedback. Unit tests are found within the system’s repository and can be set to fail a build if any of the tests fail. This prevents them from being ignored if failing, much to the contrary of tests up high in the Pyramid.

The middle layer provides verification of services and APIs, as well as the interaction between them. The amount of effort should be less than unit tests since a level of logic is ‘hidden’, therefore finding the root cause of a bug will take longer with a failed verification from this level.

Finally, on top of this structure, we have the UI tests. These tests are the closest to the actual users, and therefore cannot be defined as redundant or worthless. UI tests give a concrete piece of mind when an actual user journey is verified to be working as expected. However, as mentioned previously, going up the Pyramid means also that the ‘time to root cause’ is longer, as there is a lot of logic that needs to be dug up. A set of good UI tests can be hard to maintain unless extra effort is spent to set up a solid and reliable testing framework.

Needless to say, this topic has been blogged and discussed countless times, so my input on it is definitely not about questioning its validity. Different agile teams may have different priorities or responsibilities towards a shared application. Therefore, saying that a universal test pyramid is suitable for all the teams is quite misleading. Each team would have a different opinion on where its testing efforts are being spent, and this would reflect on their idea of the right pyramid.

“Is one pyramid not enough?”

Not really. For instance, we can think of an agile team which produced most of its commitment towards creating integrations with payment gateways. Its client would then be a team which build the front end part so that these payment methods can be consumed by the end users. We can agree that their ideal test pyramid would include a large number of unit tests, followed by a set of service layer/integration tests. Going up the pyramid, they would not be so interested in UI tests as this is not part of their own work. Consequently, they do not hold the right knowledge to design these automated tests.

Likewise, the team which would build up the visual aspect of the payment methods would spend an amount of its automation effort to verify the correctness of its UI for any regressions.

“So? Each one his own pyramid?”

No that’s not what I mean. I believe that within an organization, communication is key to everything, and this is true as well in respect to this topic. Testers and developers working in different areas of a common application should discuss and build up the pyramid of test automation together, with each team covering the right areas of the system. The responsible teams for each area hold the best knowledge for each decision, and should not be overruled by other QA members working within other parts of the system.

Conclusion

There is still a lot of speculation about how the slaves built the actual Pyramids of Egypt. What’s for sure is that they built it just how the Pharaoh wanted it to be. Make sure that when you build YOUR Pyramid, you build it how YOU need it to be, not how others in your organization suggest it to be.The more information you have on what you need to cover, the better you are at designing your own Test Pyramid.

Originally published at https://testautonation.com in 01/2018.

The Test Pyramid: Greatest Wonder of the (Software Testing) World was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.

New Year, New Framework

In the last couple of years, we have seen a boom of Node.js browser automation libraries using the Webdriver protocol. WebdriverIO is one of those libraries which has seen its popularity growing progressively. You can visualize this through the npm trends graph (you can learn more about npm trends in this post) showing the number of downloads up to two years ago.

The popularity of this library is due to some of its excellent characteristics which make it compatible with most test frameworks. It also makes it reasonably easy to jump on the ‘automation wagon’ for the newbies out there.

Quickly!! Set it up!

Let’s see how you could set up, to begin exploring the many features of WebdriverIO.

First off, prepare your framework root directory by creating a folder and go to it. Then you can initialise your Node.js framework by running the command:

You will need to enter some data for package name, version and description, or just leave them as defaults. Confirm your entries by entering yes when prompted.

Now we can go ahead and install WebdriverIO. This will install WebdriverIO as well as its dependencies and include WebdriverIO as a dependency of the framework.

npm install webdriverio

Once done, we can launch the WebdriverIO configuration helper:

./node_modules/.bin/wdio config

This will show multiple options, so let’s start with the easiest ones. Press Enter on On my local machine for the first question. The second one allows you to choose a test runner from different frameworks. The listed ones are all quite used, with a lot of similarities in their way of usage. However, a quick search on npm trends shows mocha as the most downloaded. We can discuss this on a different topic, so for now, let’s go with mocha. The next question asks you if you want the configuration tool to install the chosen framework automatically, so let’s type Y. We can now select the location of our tests, let’s go with the default one by just pressing Enter. Now we are presented with a list of reporters to display the outcome of our tests. As you can see, we are provided with a vast number of reporters which can either just output result in the terminal, as well as sending results to known CI and test case management tools such as TeamCity and TestRail.

Half way there…

WebdriverIO also gives us the choice of using multiple reporters and therefore outputting our results to different locations. For our basic example, we can go with the spec reporter by going down with the Down Arrow button and pressing Space to choose the reporter. Enter to confirm your choice. Press Enter for the next question so that this reporter is automatically installed. For the next option, we can install services which are necessary, such as ChromeDriver to run the tests using the Chrome browser and selenium-standalone to interpret our commands through the Selenium API. So let’s click the Space button on selenium-standalone and confirm using Enter. Press Enter to confirm automatic installation. For the level of verbosity, we can go with the error so as to help us when having issues. When our tests fail, the framework generates a screenshot showing the browser at the point of failure. Let’s go with the choice of default location suggested by the helper by pressing Enter. Lastly, we can keep the base URL pointing to localhost by again pressing Enter.

Done! Let’s start automating

Wait wait, let’s check that all the required packages are installed, shall we? First off, let’s get the packages currently required within the package.json file by running:

npm install

Then, we can make sure that mocha, our test runner, is also installed by running:

./node_modules/mocha/bin/mocha

This will output that no test files found. That’s okay, it means that mocha is all set within our project. Next, we need to check that selenium-standalone is available and use it to install the required ChromeDriver. Let’s do that by running:

./node_modules/selenium-standalone/bin/selenium-standalone install

If everything is fine, this will install Selenium standalone, ChromeDriver and GeckoDriver.

Once completing our checks and required installations, we can proceed with writing our first test. Next thing to do is open the project in your preferred IDE, so for this example, I will use the free text editor, Atom.

Let’s do a simple scenario where we need to verify that the Webdriver.IO navigational links are working as they should.

First, we need to create a spec file which will contain our test scenario. Create the folder structure and a file called navigationLinksTest.js under ./test/specs folder. We will create four simple tests which check that each navigational link takes the user to the expected URL. We will have three steps in each test:

Setup — Opening the browser to a given URL (the Given step in Gherkin syntax)

Action — Clicking the link that needs to test (the When step in Gherkin syntax)

Assert — Assert that the actual URL the test landed on is the expected one (the Then step in Gherkin syntax)

Talking about assertions, mocha does not come with an assertion library out of the box, so what we will do is require the assert library within the spec file. So, have a look at the code below:

var assert = require('assert');

describe('WebdriverIO Navigation Links', function() {
    it('should go to Developer Guide page when choosing Developer Guide link', function () {
        browser.url(https://medium.com/'http://webdriver.io');
        browser.click('=Developer Guide');
        assert.equal(browser.getUrl(), 'http://webdriver.io/guide.html');
    });

    it('should go to API page when choosing API link', function () {
        browser.url(https://medium.com/'http://webdriver.io');
        browser.click('=API');
        assert.equal(browser.getUrl(), 'http://webdriver.io/api.html');
    });

    it('should go to Contribute page when choosing Contribute link', function () {
        browser.url(https://medium.com/'http://webdriver.io');
        browser.click('=Contribute');
        assert.equal(browser.getUrl(), 'http://webdriver.io/contribute.html');
    });

    it('should go to Home page when choosing Home link', function () {
        browser.url(https://medium.com/'http://webdriver.io/api.html');
        browser.click('=Home');
        assert.equal(browser.getUrl(), 'http://webdriver.io/');
    });

});

By default, the WebdriverIO configuration file sets Firefox as the browser, so you need to change this to Chrome by setting this line inside wdio.conf.js to:

browserName: 'Chrome'

That’s it! Now save the file and run tests by running:

./node_modules/.bin/wdio wdio.conf.js

Voila! Your first WebdriverIO tests!

…and they lived happily ever after

This simple example of using WebdriverIO for your functional tests is far from being perfect. First of all, it follows no design pattern (we will probably discuss this in future posts), and secondly, it has no complexity involved, of which you will undoubtedly face once you need to prepare more challenging test scenarios. It is fair to say that no test framework provides an entirely stress-free experience of creating automated tests, this is no fairy tale story here. Things will get challenging, tricky, but the answer is out there. When you face a hurdle, consult with WebdriverIO’s official page as well as the usual StackOverflow. Practice, plan your tests, learn from your mistakes, learn from other testers’ mistakes, and you will succeed in making the best use out of this great tool!

Want to share your thought about WebdriverIO? Leave us a comment below!

P.S. If you want to have a look at the code, you can do so by going to https://github.com/testautonation/Webdriverio-intro

Originally published at https://testautonation.com in 01/2018.

Tester, I introduce to you WebdriverIO was originally published in TestAutonation on Medium, where people are continuing the conversation by highlighting and responding to this story.