Distributing Robot Framework Test Suites Evenly with SuiteVisitor and Bin Packing

One of the recurring problems in test automation is how to run hundreds or thousands of test cases through a CI pipeline efficiently. Test cases — or whole test suites — can be split across workers statically or automatically, based on measured cost. This article covers the automatic approach: assigning test suites to workers by implementing the visitor pattern with Robot Framework’s APIs.

As an example, consider the case of an automation with a relatively small number of test cases. This particular CI pipeline executes jobs of about 621 end-to-end (E2E) test cases, at least once daily.

As shown in the histogram below (Figure 1), the “average” test case takes 93.7 seconds, while the shortest takes just 15 seconds and the longest over 7 minutes 40 seconds. Most of the test cases — 90% of them (P90) — execute in less than 170 seconds.

Figure 1. Histogram of Test Case Counts by Execution Time

One worker executes all test cases in about 970 minutes (~16 hours) and by dividing the work among a number of workers, seven workers in this case, the average elapsed time is 138 minutes per worker.

In this implementation of Robot’s SuiteVisitor with Bin Packing, test cases are executed within test suites with the actual elapsed time per worker approximating the average elapsed time at 146 minutes. Figure 2 below shows time for each worker, VM1 to VM7.

Figure 2. Elapsed Time per Worker (VM1-VM7)

The gap between the 138-minute theoretical average and the 146-minute actual average comes from what the algorithm actually packs: whole test suites, not individual test cases. This run's 621 tests are organized into 52 suites, and those suites carry 991.1 minutes of combined Robot-reported elapsed time — including each suite's own setup and teardown — versus the 970 minutes implied by summing raw individual test durations. Figure 3 shows how unevenly that suite-level time is distributed, which is exactly the imbalance to_constant_bin_number has to work around (line 25, Figure 4).

Figure 3. Suite Duration Distribution

The following code implements Robot Framework API:

Figure 4. Visitor Implementation with Bin Packing

Walking through SplitSuite

from robot.api import SuiteVisitor
from robot.api.logger import console
import binpacking

class SplitSuite(SuiteVisitor):
    """Visitor that divides the visited suite structure in parts of equal size."""
    def __init__(self, parts, which_part=1):
        self.parts = int(parts)
        self.which_part = int(which_part)

SplitSuite extends Robot Framework's SuiteVisitor — the same interface Robot itself uses internally to walk the suite tree before execution. Because it's registered as a pre-run modifier (--prerunmodifier), Robot hands it the entire suite tree before a single test runs, and whatever the visitor leaves behind is what actually executes. Two arguments come in from the command line: parts, the total number of workers, and which_part, which worker this particular Robot process is. Robot passes these straight from the modifier string, colon-separated:

--prerunmodifier ui.splitsuite_visitor.SplitSuite:7:3

reads as “split into 7 parts, and this process runs part 3.”

def visit_suite(self, suite):
        dict_of_suites = {}
        for s in suite.suites:
            s.filter(excluded_tags=['archived', 'defect-failure'])
            console(f'Test suite name, number of test cases: {s.name, s.test_count}')
            dict_of_suites[s] = s.test_count

visit_suite is the one method that matters here — Robot calls it once for the root suite. For each test suite (each test folder), it first filters out tests carrying tags we never want to run in CI — archived, defect-failure — before counting, so a suite with 40 tests but 15 archived contributes a weight of 25, not 40. That filtered test_count becomes the weight in a dictionary keyed by the suite object itself:

Get Luis A Gomez-Tinoco, MS(CS)’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

{suite_object: weight}.

bins = binpacking.to_constant_bin_number(dict_of_suites, self.parts)
client_bin = bins[self.which_part - 1]

This is the actual bin-packing step, and it’s a one-line call into the binpacking package. to_constant_bin_number takes that {suite: weight} dict and greedily distributes the keys (whole suites — a suite is never split across workers) into exactly self.parts bins, trying to keep each bin's total weight as close to equal as possible. It returns a list of dicts, one per bin. self.which_part - 1 picks out this worker's bin (converting from Robot's 1-indexed which_part to a Python list index).

Why to_constant_bin_number, not to_constant_volume

The binpacking package's other algorithm, to_constant_volume, solves a different problem: given a maximum capacity per bin, use as few bins as possible. That's the right tool when the number of workers is flexible and you're trying to minimize infrastructure — e.g., "however many VMs it takes to keep each one under 3 hours." But in this pipeline, the number of workers isn't a variable to optimize — it's fixed by however many Jenkins agents are actually provisioned and available for the run.

list_of_suites = list(client_bin.keys())
suite.suites = list_of_suites

The last line is the payoff: suite.suites is the actual list Robot Framework will execute. Everything not selected for this worker's bin is simply never assigned back — it's dropped from the tree for this process. Multiply this by the number of parallel Jenkins agents, each running the identical command with a different which_part, and the full 621-test suite gets partitioned across all of them with no test running twice and no coordination between workers beyond the shared, deterministic weight dictionary each one independently reconstructs and bins the same way.

Takeaways

The core idea here isn’t specific to Robot Framework or even to testing — it’s a general pattern for parallelizing any fixed set of independent, variable-cost units of work across a fixed number of workers: measure the real cost of each unit (not just its count), then let a bin-packing algorithm do the balancing instead of splitting alphabetically or by folder count.

In this pipeline, that one-line call to to_constant_bin_number is what turns a nearly 16-hour sequential run into a ~3-hour parallel one across 7 VMs. The remaining imbalance — 1 hour 51 minutes on the fastest worker (vm2) versus 3 hours 5 minutes on the slowest (vm4) — is the visible cost of the one simplification the algorithm makes: it packs whole suites, not individual tests, so a single large suite can still tip a bin's weight.

A logical next step would be splitting at the test-case level rather than the suite level, at the cost of losing suite-level setup/teardown reuse between tests. For now, whole-suite bin packing gets most of the benefit for very little code — one SuiteVisitor, one library call.