Mutation Testing?

While I’m reading “effective testing” book, I found the word that is “mutation testing”. I was wondering about the meaning of it, and I immediately searched it.

This way of testing is so mind-blowing for me. I’ve never heard about it before. The way of testing is making mutation for checking our test-system to check exceptional cases as well. It’s too long to write in Summary. So, Let’s dive into what “mutation testing” is!

Mutation Testing

As I said earlier, mutation testing change source code (in Pitest for Java manipulate java-bytecode). We call this action “mutate”. For example, there are some operations that mutate our code.

if (a < b) {
  // do something
}

After mutation, this code would be changed to “≤” like the below one.

if (a <= b) {
  // do something
}

There are this kind of operation that change our code. You may have questions like “what mutation testing for?”. So, I’ll show you some simple example to show the purpose of mutation testing.

Mutation Testing in practice

In this article, I’m gonna use “kotiln”, “kotest”, and “pitest”. So, If you are not familiar with these stuff. Then you just clone my repository and execut it. It would be enough to understand the concept of “mutation testing”

Get Roach’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Let’s say there are developer who is called by “roach” (is his a.k.a). Roach want to get off work after finishing his last task. His last work is just writing test code to verify the below function.

package com.example

class NCondition {

    fun fooIfIsNumberIsPositive(number: Int): String =
        if (number > 0) "foo" else "bar"
}

As saw this function, roach said, like “Oh, It’s a piece of cake!!”, and wrote the test code like the below one.

package com.example

import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe

class NConditionTest : FunSpec({

    context("fooIfIsNumberIsPositive") {
        test("should return foo if number is positive") {
            NCondition().fooIfIsNumberIsPositive(5) shouldBe "foo"
        }

        test("should return bar if number is zero") {
            NCondition().fooIfIsNumberIsPositive(-1) shouldBe "bar"
        }
    }
})

It seems to be nice and concise test code for verifying the function, and roach thought of the same as we think. Before getting off work, Roach just executed pitest just in case.

And the report from pitest told that “your test case is not enough for detecting changes! One mutation survived, even though, it was executed by test case in our system!!”

Roach composed himself again, and looked at the report again. The report told roach, it’s success even we changed your conditional boundary(`>` to `=>`) to the other boundary, then your test case is success!

So Roach thought about what would happen if the conditional boundary changed. Roach just came up with the phrase from this book, which was “you always should pay attention to the boundary value”. After while, roach realized that he should write test code for “zero”. As soon as he wrote test code, and executed it, the mutation died!

In conclusion

As you saw this article, the mutation testing can help you to find the exceptional case, and make your testing-system more robust based on the result. But, That doesn’t mean it’s a perfect fit everywhere. You should be more careful, if you want to put this one into your project. You should think about what you can get from the result, like this.

If you want to talk about the advantages of using this kind of mutation testing on production level, please feel free to DM me. I’d like to chat with you guys who are interested in this topic.

Linked-In: https://www.linkedin.com/in/%EC%8A%B9%ED%98%84-%EC%A0%95-376842221/?locale=en_US