Performance Testing Tools: What I’d Actually Look At Before Choosing One
Have you seen an application that works perfectly during testing but significantly underperforms when exposed to actual users? This has been seen several times before. When there is a low number of users, everything looks fine — pages load, APIs react, transactions go through.
But when more users try to use the application, its performance drops — response time grows, requests start timing out, and there are more errors. The first question that usually arises is “Why wasn’t it seen during testing?” And this is what supports the position that performance testing should be taken seriously.
Talking about performance testing tools, one can get easily confused with all the options available. There are separate tools for load testing, stress testing, API performance testing, big enterprise testing, cloud-based testing, etc. Instead of looking for the “best” tool, I decided to take another approach — what kind of tool will actually help to see how does an application behave under load?
The First Thing I’d Want to Know
Before choosing a tool, it is crucial to understand the application that is being analyzed. What tasks do people usually perform? For instance, a possible user story can look like this: Login → Search → Choose Product → Add to Cart → Purchase. This is more useful than a rather vague suggestion like “test 10,000 users,” because there is no point in testing a workload of 10,000 people who perform no actions at all. It is crucial to find out what kind of user activities are done and how often.
In this case, performance testing tools (JMeter, LoadRunner, Gatling, BlazeMeter, NeoLoad, and WebLOAD) start to make sense.
There are many different techniques that help to create and execute workloads. However, the relevance of such tools depends on the correctness of the workload.
JMeter Is Still Hard to Ignore
In terms of performance testing, I always begin with considering JMeter first as a suitable tool. The reason behind this choice is the open-source nature of JMeter. This, however, is not enough to ensure the effectiveness of a tool.
The main issue from my point of view is the flexibility, because with the help of it you can create a realistic scenario and model requests according to that scenario rather than just creating traffic for your application. For example, you can model the scenario where your users will authenticate, do some searching, call an API and finally make some transaction.
This kind of scenario will produce some meaningful metrics. The second important point is the ability to gradually increase the load. Start from a low load, analyze the behavior of the system, increase the number of users, look at response times and errors and so on until you see that the system starts failing.
Then There Are Enterprise-Focused Tools
As far as needs of a big enterprise application go, there can be great differences. In such cases, there is room to consider using products such as LoadRunner and NeoLoad.
While the performance testing of an enterprise application can involve more than one website, involving other things such as API calls, database interaction, different services, authentication, integration with external resources, different components that together make up one business transaction, in such cases it is better to have a performance testing tool that will handle the complexity of the environment rather than force you to simplify your application to test it.
What About Cloud-Based Performance Testing?
Another one is of considerable interest. Although there is a possibility to build performance testing on the laptop, it doesn’t mean that this preparation can guarantee the capability to make realistic, big traffic from the same laptop. Cloud solutions come handy here.
As an example, we can take BlazeMeter for the purpose of scaling up performance testing as well as running tests in more distributed manner. Yet, it is worth giving a warning. More load doesn’t guarantee better test. If the test environment is not similar to the real one, it will be deceitful anyway. There is the possibility to make big traffic and get some insights about how the real users will work with the application.
Gatling Takes a Different Approach
The Gatling framework starts becoming useful when the team feels confident about using a more programming-oriented way of doing performance testing.
This works especially well in terms of incorporating performance testing in the development process and CI/CD process, as opposed to limiting its use to the late stage of a release.
In my opinion, this represents a major change. Performance testing should not be viewed as a chain of actions: build the application → finish developing it → conduct performance testing → discover problems.
Performance testing problems need to be discovered as early as possible. The more convenient it is to incorporate performance testing into the engineering process, the better.
WebLOAD and the Question of Scale
WebLOAD is another solution to take into account in order to resolve more complicated or bigger testing needs. Nevertheless, when we talk about scale, I am not too concerned about how many virtual users a particular tool can create.
On the contrary, I ask myself the following questions:
Is it possible to create a realistic load?
Is it possible to control user’s behavior?
Can the same scenario be repeated?
Is it possible to understand what has happened in the course of the test?
Where is the bottleneck?
These questions provide me with more useful information.
Here’s Where Performance Testing Gets Interesting
Get Sonalsrawlekar’s stories in your inbox
Join Medium for free to get updates from this writer.
Performance testing is relatively easy to carry out; the problem is the analysis of the data obtained from this test. Assume that the test is finished and has produced the following figures:
- Average response time: 2 seconds
This is good or not? It might be or might be not. To make more in-depth assessment, one needs to know the following:
- 95th percentile response time
- Slowest response time
- Number of failures
- Were response times rising as the user load increased?
- Was the CPU utilization unusually high?
- Was the database a bottleneck?
- Was memory consumption increasing?
- Could the system cope as the load decreased?
Thus, performance testing should not end with the statement “Test passed.” The real value is to understand why the system behaved in this way.
And This Is Where Different Types of Testing Come In
One additional aspect is that performance testing does not consist only of load testing. For example:
- Load testing determines the application’s behavior in terms of expected loads.
- Stress testing determines the behavior of the system under pressure above the designed capacity.
- Spike testing determines how the system reacts to rapid spikes or drops in loads.
- Endurance testing determines the performance under load for a long period of time.
- Volume testing determines the behavior of the application under big volumes of data.
The chances that one test will cover all the above areas are very low. First, it is necessary to determine what aspects of the application need to be tested and then to develop appropriate tests.
The Biggest Mistake I’d Avoid
One should not assume that a performance test is helpful only on account of creating significant traffic.
An example would be an e-commerce application under a 10,000 user traffic load, all of whom are carrying out identical operations simultaneously.
This does not represent actual user behavior in any way.
The truth of the matter is that users carry out different kinds of tasks; some search, some login, some put items into their shopping cart, some complete transactions, some leave before completing transactions, etc.
A realistic workload therefore adds more value than simply having traffic.
What I Actually Want From a Performance Testing Tool
Having analyzed different approaches to performance testing, I came to the conclusion that the tool per se is not the most important factor.
What really matters is what insights can be gained with the help of the tool. It is important to have the ability to create realistic workloads, to be able to repeat the test, scale the load if necessary, and collect the right metrics.
However, the most important thing about the tool is that it should give enough information for the analysis of the reasons for performance degradation.
My Final Thought
While performance testing is not about the query “How many users can my application handle?” but rather about a much more complex question “How will my application perform under the real user load?”, there are tools that will give the answer to this question, and there are numerous options: JMeter, LoadRunner, Gatling, BlazeMeter, NeoLoad, WebLOAD, etc. Nevertheless, choosing the right tool is not about choosing the tool with the most extensive capabilities and highest numbers of simulated users.
The correct workflow starts with the application, and this means defining the user journey, creating the real workload, conducting the test and gradually increasing the load until the breaking point is found.
It is important to mention that performance testing should not be about just applying the load; on the contrary, it should be about finding the point of degradation and understanding the reasons for this degradation
