Create Powerful Tests using Property Based Testing

Photo by RetroSupply on Unsplash

Create Powerful Tests using Property Based Testing

1. Introduction

Suppose you have developed a bus passenger management application and you would like to write some automated tests to check if your code is robust.

The Bus class looks like this

data class Bus(val code: String, val maxCapacity: Int = 20) {

    private val passengers: MutableList<Passenger> = mutableListOf()

    init {
        require(code.isNotBlank()) {
            println("The code must not be blank")
        }
    }
    fun addPassenger(passenger: Passenger) {
        val emailExist = passengers.any {
            it.emailAddress == passenger.emailAddress
        }
        require(!emailExist) {
            val message = "This email (${passenger.emailAddress}) address is already used !"
            throw IllegalArgumentException(message)
        }
        require(passengers.size < maxCapacity) {
            val message = "This bus only support $maxCapacity passengers !"
            throw IllegalArgumentException(message)
        }
        passengers.add(passenger)
    }

    fun isBoarded(passenger: Passenger) = passengers.contains(passenger)
}

The bus has a code and a capacity that defines the number of passengers that the bus can take, when creating the bus, the code must not be empty, and when adding a new passenger we make sure that the passenger's email is unique and that we have not yet reached the limit of passenger numbers.

And the passenger class is quite simple

data class Passenger(val name: String, val emailAddress: String) {
    init {
        require(name.isNotBlank()) {
            val message = "The passenger name should not be blank"
            throw IllegalArgumentException(message)
        }

        val emailRegex = """^(.+)@(\\S+)${'$'}""".toRegex()
        require(!emailRegex.matches(emailAddress)) {
            val message = "This email address is not valid !"
            throw IllegalArgumentException(message)
        }
    }
}

The logic of the passenger class is just as simple as the previous class, when creating a passenger we have to make sure that he has not entered an invalid name and email address.

1.1 Example Based Testing

You have created the components of your program and you are happy, now you would like to test adding passengers to the bus by creating some concrete example of passengers, the code to do this can look like this

class BusManagementTest {

    @Test
    fun `should add some passenger`() {
        val bus = Bus(code = "BUS-001")

        val derrick = Passenger(name = "Derrick", "derrick@sample.com")
        val john = Passenger(name = "John", "john@sample.com")
        val sandra = Passenger(name = "Sandra", "sandra@sample.com")

        bus.addPassenger(derrick)
        bus.addPassenger(john)
        bus.addPassenger(sandra)

        Assertions.assertTrue(bus.isBoarded(derrick))
        Assertions.assertTrue(bus.isBoarded(sandra))
        Assertions.assertTrue(bus.isBoarded(john))
    }
}

The above code snippet is very simple, we create the bus and some passengers, we add the three passengers in our bus and in order to check if they are really in the bus, By running this test everything should pass without any problem and you are happy

But the question is, what is we didn't used to right sample to exercice our program ? Does it not possible to make the testing framework to choose several concrete sample for us and make our test less dependent to concrete sample we create manually ?

This is what property-based testing is about. We do not pick a concrete example; rather, we define a property (or a set of properties) that the program should adhere to, and the test framework tries to find a counterexample that causes the program to break with these properties.

1.2. Example Based Testing Downside

Besides the fact that the above test is highly dependent on the samples we created manually, if you analyze it very well you will notice some problems.

  • Documentation :

A test is supposed to document the code it is testing, but in the previous case we notice that the Test doesn't tell us much about the code we are testing.

  • Input :

To ensure a code is robust it must be tested with a large amount of input data, but in our case we just test with 3 passengers, it's true that we could do better by adding more passengers it can be solution but that'll solve the problem partially and we will see how shorty.

  • Boundaries :

A test should show us the boundaries of the program we are testing, for example we can't add a number of passengers that exceeds the capacity of the bus, the email address of each passenger must be unique etc., looking at our test it is really hard to see the boundaries of our program

It's true that we could make the previous test a little more interesting by adding more test cases, but the problem is that it will require more code.

So how can we make our test a little more powerful by testing our program with a variety of input data while showing us the boundaries of our program?

This is where Property Based Testing comes in.

3. Property Based Testing

3.1. What's Property Based Testing

Developers typically write example-based tests. These are your garden variety unit tests you know and love. You provide some inputs, and some expected results, and a test framework like Kotest or JUnit checks that the actual results meet the expectations.

One problem with this approach is that it is very easy to miss errors due to edge cases that the developer didn't think about, or lack of coverage in the chosen inputs. Instead, with property testing, hundreds or thousands of values are fed into the same test, and the values are (usually) randomly generated by your property test framework.

For example, a good property test framework will include values like negative infinity, empty lists, strings with non-ascii characters, and so on. Things we often forget about when writing example based tests.

You can use property based testing everywhere where a wide range of input data can be an useful.

3.2. Property Based Testing in Action

In order to convert our Sample Based Testing code into Property Based Test, we are going to use Kotest, Kotest is a testing framework for Kotlin that support Property Based Testing, but don't worry property based is supposed by several testing framework in the case Kotlin is not your main programming language.

3.3. Installation

In order to use Kotest you need the kites-property dependency, Kotest is a Kotlin Multiplatform library so you have to choose the right dependency accordingly to your target.

dependencies {
   testImplementation("io.kotest:kotest-property:$version")
}

3.4. Kotest Property Based Test

Here is the modified version of our previous test, the syntax seems a bit weird, to know more I suggest you to have a look on the different styles used by Kotest

class BusManagementTest : StringSpec() {

    init {
        "should add some passenger" {
            checkAll (
                iterations = 100, 
                genA = Arb.string(), 
                genB = Arb.list(gen = Arb.string(), range = 100..100),
                genC = Arb.list(gen = Arb.string(), range = 100..100)
            ) { busCode, names, emails ->

                val bus = Bus(code = busCode)
                names.zip(emails).map { entry ->
                    Passenger(name = entry.first, emailAddress = entry.second)
                }.forEach { passenger ->
                    bus.addPassenger(passenger)
                    bus.isBoarded(passenger) shouldBe true
                }
            }
        }
    }
}

I will try to explain line by line to help you understand what this code is trying to do.

  • To generate the values we will have to use the checkAll method with some parameters, the first parameter of the checkAll method defines the numbers of the values that will be generated in our case 100, and the 3 other parameters are generators (Arb.x()), the first one will generate a string and the two other generators will generate lists

  • The generators of the list Arb.list() takes as parameter another generator defining the type of the content of the list which will be generated, the size of the generated list will be pickup random from the range provided as the second parameter and in our case as the min and max value of the range are equal it means the generated list will be of 100 elements if not the two generated list can have different sizes.

  • ThecheckAll method also takes as last parameter a lambda, the checkAll method will call this lambda with the generated values according to the generators used.

  • Once we have all the necessary values to create the bus and its passengers, we create the bus then we add passengers, then we check if the passenger is really in the bus with the instruction bus.isBoarded(passenger) shouldBe true

To read more about all those function check out this list from the Kotest documentation.

3.4. Running the Test

Our test was an Example Based Test but now it's a Property Based Test, now let's try to run the test to see what happens.

Screen Shot 2022-05-18 at 01.03.08.png

When running this test you will notice that the test did not pass, and the error message says that the passenger name and bus code must not be empty strings.

With property based testing we are able to test our application with several test cases, test cases that could easily be forgotten in the first version of your test are easily taken into account in the new version.

Now let's try to modify the test so that we no longer have empty strings

class BusManagementTest : StringSpec() {

    init {
        "should add some passenger" {
            checkAll (
                iterations = 100, 
                genA = Arb.string(minSize = 1),
                genB = Arb.list(gen = Arb.string(minSize = 1), range = 100..100),
                genC = Arb.list(gen = Arb.string(minSize = 1), range = 100..100)
            ) { busCode, names, emails ->

                val bus = Bus(code = busCode)
                names.zip(emails).map { entry ->
                    Passenger(name = entry.first, emailAddress = entry.second)
                }.forEach { passenger ->
                    bus.addPassenger(passenger)
                    bus.isBoarded(passenger) shouldBe true
                }
            }
        }
    }
}

To tell Kotest to stop generating empty strings, just specify the minimum size of the string to be generated as in the code above.

Let's Run the test again to see what happens.

Screen Shot 2022-05-18 at 01.11.53.png

As you can see in the above screenshot, the error message is different saying that no more than 20 passangers can be added to a bus

Let's try to modify our test again to generate only 20 passengers by modifying the from 100..100 to 20..20 like in the following code.

class BusManagementTest : StringSpec() {

    init {
        "should add some passenger" {
            checkAll (
                iterations = 100, 
                genA = Arb.string(minSize = 1),
                genB = Arb.list(gen = Arb.string(minSize = 1), range = 20..20),
                genC = Arb.list(gen = Arb.string(minSize = 1), range = 20..20)
            ) { busCode, names, emails ->

                val bus = Bus(code = busCode)
                names.zip(emails).map { entry ->
                    Passenger(name = entry.first, emailAddress = entry.second)
                }.forEach { passenger ->
                    bus.addPassenger(passenger)
                    bus.isBoarded(passenger) shouldBe true
                }
            }
        }
    }
}

Screen Shot 2022-05-18 at 01.14.19.png

After running the test this time the error message says that we tried to register a passenger with a mail address that already exists

Let's try again to modify the code to generate unique email addresses.

class BusManagementTest : StringSpec() {

    init {
        "should add some passenger" {
            checkAll (
                iterations = 100, 
                genA = Arb.string(minSize = 1),
                genB = Arb.set(gen = Arb.string(minSize = 1), range = 20..20),
                genC = Arb.set(gen = Arb.string(minSize = 1), range = 20..20)
            ) { busCode, names, emails ->

                val bus = Bus(code = busCode)
                names.zip(emails).map { entry ->
                    Passenger(name = entry.first, emailAddress = entry.second)
                }.forEach { passenger ->
                    bus.addPassenger(passenger)
                    bus.isBoarded(passenger) shouldBe true
                }
            }
        }
    }
}

To generate lists that do not contain duplicates instead of using Arb.list() we use Arb.set() instead, if you run the test again it will not pass as the email address is not valid, we can ask Kotest to generate valid email addresses as the following code shows, we used Arb.email() instead of Arb.string()

class BusManagementTest : StringSpec() {

    init {
        "should add some passenger" {
            checkAll (
                iterations = 100, 
                genA = Arb.string(minSize = 1),
                genB = Arb.set(gen = Arb.string(minSize = 1), range = 20..20),
                genC = Arb.set(gen = Arb.email(), range = 20..20)
            ) { busCode, names, emails ->

                val bus = Bus(code = busCode)
                names.zip(emails).map { entry ->
                    Passenger(name = entry.first, emailAddress = entry.second)
                }.forEach { passenger ->
                    bus.addPassenger(passenger)
                    bus.isBoarded(passenger) shouldBe true
                }
            }
        }
    }
}

Now if you run the test, everything passes without problems.

Screen Shot 2022-05-18 at 01.23.53.png

In the end we end up with a robust test, which documents the code, anyone reading this test will understand that the bus code and the passenger name must be at least one character long, the email to create the passenger must be valid and unique and that the number of passengers must not exceed 20.

4. Conclusion

Property Based Testing is a very interesting approach allowing us to create robust tests compared to Example Based Testing, test cases that can easily be forgotten when doing Example Based Testing are easily taken into account with Property Based Testing

There are several testing frameworks that allow Property Based Testing Kotest is not the only one

If you have any questions please feel free to contact me on twitter.

5. Read More

Did you find this article valuable?

Support Eric Ampire by becoming a sponsor. Any amount is appreciated!