This article is my personal opinion, and I’m happy to hear your thoughts about this topic.

Nowadays, more and more companies start to adopt Agile practices to transform the organization’s structure to perform in a flexible and fast-changing environment efficiently. This transformation usually involves the restructuring of the existing roles and departments.

I was working in several software companies that decided to move from separate developers/QA/designers departments to Scrum teams (Feature teams). This article is my personal experience and thoughts on a QA manager role in Scrum teams. I was lucky to be on both sides during these transformations — as a QA engineer and as a QA department lead. It helps me to assess the situation from different perspectives.

First of all, let’s recall the classical QA manager job requirements:

• manage the QA team
• communicate with other departments to coordinate QA efforts
• take responsibility for the product quality
• establish QA policies and practices across the QA team
• monitor and report about QA team activities

On the other hand, in Scrum (one of the Agile frameworks), we can differentiate the following roles:

• Scrum Master
• Product Owner
• Scrum Team
• Stakeholders

The Scrum team, in particular, includes a mixture of competencies, e.g., developers, QA engineers, designers, analysts, etc. The core features of the Scrum team is that it is self-organized and cross-functional:

• Self-organized: decide how to work without any assistance from non-team members
• Cross-functional: have the necessary skills to complete the work without depending on non-team members

In some companies, the Scrum team has a formal or informal Team lead who acts as a team coordinator and a team representative. We call this role variously: Team lead, Technical lead, Delivery manager, Engineering manager. Usually, it’s a formal role for companies that can’t give up the idea that each employee must have a manager who tracks the employee’s performance and gives approvals for a vacation :)

This article is not about Scrum, so I’m not going to describe each role in detail. One of the main points is that there is no dedicated QA team in a Scrum environment, so there is no QA Manager role.

Typically, the QA Manager role stays in a company as an echo of the old structure. In the best-case scenario, this role can be transformed into a QA Mentor or QA Architect. But frequently, the QA manager stays isolated from the product development process and, in the worst-case scenario, attempts to continue their past activities, making a lot of unforgivable mistakes.

Continue managing

As far as QA specialists are the part of the Scrum team, they are not supposed to report to a QA manager who is not involved in team activities. All their tasks and efforts should be coordinated with Scrum team members. Their performance should be assessed in terms of achieving team goals. Thus there is no sense to set separate QA goals that are not correlated with Scrum team activities and objectives.

Summary: The QA Manager should stop managing and start mentoring in case the Scrum team asks for assistance from a highly qualified QA professional. Undoubtedly, assigning tasks directly to QA engineers in Scrum teams without communication with PO, Scrum Master, and Team lead — is the common mistake. This approach undermines the idea of Scrum since all tasks must be prioritized and evaluated by the team.

Tracking all Scrum teams activities

When the company decides to break down the structure into multiple small Scrum teams, it becomes almost impossible to participate in all their activities. There are QA Managers who try to involve themselves in all the meetings across all Scrum teams: grooming & planning sessions, dailies, retrospectives, etc. From first glance, their calendar shows how busy they are, and it can be the evidence that their role is essential and irreplaceable. But in reality, there is no sense to follow up with all teams, because they are intended to be self-sufficient. Of course, the QA manager can be helpful in some cases in attempts to coordinate the multiple teams. However, it will likely lead to the result that teams will start losing their self-organization and giving part of the responsibility to a non-team member, which discredits the sense of Scrum teams.

Summary: QA Manager should start trusting their QA colleagues and get insights from them on regular but not too frequent meetings (biweekly is the excellent frequency). The QA Manager should become a facilitator on these meetings to help all QA specialists across the organization track what’s happening in other teams and how it can influence their team.

Being in charge of product releases

The Scrum team takes the responsibility from the very beginning to the last day of feature support. It means that the team makes collective decisions on the following aspects:

• the readiness of a product to release (the level of product quality is high enough to go live)
• the set of secondary release artifacts (user guides, documentation, automation, etc.)

The concept is that the Scrum team is responsible for fixing production issues and the further support and maintenance of the released feature. In that knowledge, the team decides what additional release artifacts will help to support the feature in production (test documentation, automation coverage, etc.).

Undoubtedly, the good practice is to notify all other Scrum teams about the upcoming feature release. And of course, the most challenging is to make sure the product architecture provides the possibility to deliver features independently and safely without breaking the whole system with your changes.

Summary: QA Manager should not take responsibility for a release and production bugs. Thus there is no need to get the approvals from QA Manager or any other “former structure” manager to release the feature. But what should be done — is providing Scrum teams with all necessary assistance to set up the reliable release processes. It sounds like an Architect role. Having that said, the QA Manager role can be transformed into the QA Architect role to provide the stable prod-like test environments, reliable regression test automation, and all other aspects that can reduce the possible adverse effects after the release.

Establishing “one size fits all” approach in Scrum teams

Each Scrum team should decide what practices and approaches suit their needs and goals. It’s expected that team members are highly qualified enough to be able to define strategies on how to perform their tasks. The attempts to unify all processes and apply general policies that dictate a “one size fits all” approach may result in a losing strategy.

Summary: QA Manager should not ignore the needs and decisions of QA engineers in Scrum teams. Thus all suggestions on how to unify the existing ‘zoo of tools and processes’ should be done very thoroughly and carefully. The proposed solution should bring a profit to teams, not just an additional load. Usually, the universal approach that does not take into account the interests of all interested parties will not bring any positive effect, and may even destroy what already exists and brings positive results.

Conclusion

If your company moved to Scrum teams I see the following options regarding the role of QA Manager:

1. Eliminate this role at all
2. Transform QA Manager role into QA Mentor / Facilitator, or if it’s possible — QA Architect (usually it depends on a technical background)

If your company moved to Scrum teams, but the QA manager continues acting the same way it was in the past — I have some bad news: you’re killing your Scrum.

A lot of QA engineers have Postman collections for testing API endpoints. Usually, these collections contain tests that check:

• response codes
• response body structure
• response body contents

But sometimes we need to do performance tests, e.g. check that the response time is within some baseline values. But the response time for a single request is not representative due to various reasons:

• issues with a network
• issues with a test environment
• bad luck

That’s why we used to acquire specific performance testing tools like JMeter. But what if we want to use a single tool for testing our API from a functional and performance perspective?

Let’s try to solve this problem by Postman.

Inputs

• Endpoint to test: dummy.restapiexample.com/api/v1/employees
• Number of requests: 100
• The time of delay between requests: 100 ms
• The baseline value of 90 percentile response time: 1000 ms
• Goal: create a Postman test that checks the 90 percentile response time for a specified endpoint.

What is the 90 percentile response time?

To explain this concept, I want to give an example: let’s say we have to check the response time for the request:

GET dummy.restapiexample.com/api/v1/employees

We will send this request 100 times with a 100-ms delay. We will collect the response time values for each request. Then we’ll sort all the response time values in ascending order. Afterward, we’ll take the first 90% transactions out of this set. As a result, the response time that has the maximum value among them is considered as the 90 percentile response time.

Let’s start

Create a collection with a request you want to test.

Then open the Tests tab and write the following test script:

//1000 ms is a maximum allowed value according to requirements 
maximumResponseTime = 1000;

//100 is a number of sent requests according to requirements 
iterations = 100;

//100 ms is a delay between requests according to requirements 
delay = 100;

//responseTimes is an array for collecting response time values
responseTimes = [];

i=0;
function sendRequest() {
    pm.sendRequest({
        url: "dummy.restapiexample.com/api/v1/employees",
        method: 'GET'
    }, function (err, res) {
        pm.test("Response time is " + res.responseTime, function (){
        pm.expect(err).to.equal(null);
        pm.expect(res).to.have.property('code', 200);
        responseTimes.push(res.responseTime);
        });
        if (i < iterations - 1) {
            i++;
            setTimeout(sendRequest, delay);
        } 
        else {
            percentile90ResponseTime = quantile(responseTimes, 90);
            pm.test("90 percentile response time " + percentile90ResponseTime + " is lower than " + maximumResponseTime + ", the number of iterations is " + iterations, function () {
                pm.expect(percentile90ResponseTime).to.be.below(maximumResponseTime);
            });
        }
    });
}
sendRequest();

function sortNumber(a,b) {
    return a - b;
}

function quantile(array, percentile) {
    array.sort(sortNumber);
    index = percentile/100. * (array.length-1);
    if (Math.floor(index) == index) {
     result = array[index];
    } else {
        j = Math.floor(index)
        fraction = index - j;
        result = array[j] + (array[j+1] - array[j]) * fraction;
    }
    return result;
}

In the 1st part of this script, we declare some baseline values: number of iterations, the maximum acceptable response time value and the time of delay between requests. Then we call functions that send our request for a specified number of iterations with a required delay. All the response time values are stored in responseTimes array.

The test that checks the 90 percentile response time calls a helper function called quantile(responseTimes, 90). It’s declared at the end of the script. This function uses another helper function sortNumber that is required for sorting our array in ascending order. For more information use this reference.

Results

When you hit Send the request, you can check the Console that will show the sequence of requests being sent.

The Test Results tab will show the response time records for all requests and the final test result: e.g. 90 percentile response time 808.1 is lower than 1000, the number of iterations is 100.

That’s it. You can use this script in your Postman tests and check it out. Good luck :)