Thursday, March 7, 2013

Continuous Performance and JUnit with ContiPerf

If you want to introduce some continuous performance into your Java project, you should definitively check the ContiPerf library : that tool transforms your unit tests into performance tests simply by adding few annotations !

For instance, suppose your need to do some XML serialization with JAXB as illustrated below and therefore you have to test the UserSerializer class.


The corresponding code for the UserSerializer class is:
package org.javabenchmark;

import java.io.File;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;

public class UserSerializer {

    public void serializeUserToXmlFile(User user, File file) throws JAXBException {

        JAXBContext jaxbContext = JAXBContext.newInstance(User.class);
        Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
        jaxbMarshaller.marshal(user, file);
    }
}
and the corresponding code for the UserSerializer test would be:
package org.javabenchmark;

import java.io.File;
import java.util.UUID;
import javax.xml.bind.JAXBException;
import org.junit.Test;
import static org.fest.assertions.api.Assertions.*;

public class UserSerializerTest {
    
    @Test
    public void shouldSerializeRandomUser() throws JAXBException {
        
        // given a random user
        String id = UUID.randomUUID().toString();
        String firstName = "Julien";
        String lastName = "Paoletti";
        
        User randomUser = new User();
        randomUser.setId(id);
        randomUser.setFirstName(firstName);
        randomUser.setLastName(lastName);
        
        // when serializing it to a random XML file
        File xmlFile = new File("target/" + id + ".xml");
        xmlFile.deleteOnExit();
        
        UserSerializer serializer = new UserSerializer();
        serializer.serializeUserToXmlFile(randomUser, xmlFile);
        
        // then the file contains the corresponding XML
        String validXmlContent = ""
                + ""+ firstName + ""
                + ""+ lastName + ""
                + "";
        assertThat(xmlFile).hasContent(validXmlContent);
    }
}

The test passes,  it is now time to transform this unit test into a performance one in 2 steps:
  1. add a JUnit rule from ContiPerf
  2. add a ContiPerf annotation to configure the execution of the performance test

public class UserSerializerTest {

    @Rule
    public ContiPerfRule i = new ContiPerfRule();
    
    @Test
    @PerfTest(invocations = 200, threads = 2)
    public void shouldSerializeRandomUser() throws JAXBException {

        // ...
    }
}
As you can see, the test will be exectuted 200 times by 2 threads. On my computer (Ubuntu VM 5Go 4 CPU), running the test produces the following output:

org.javabenchmark.UserSerializerTest.shouldSerializeRandomUser
samples: 200
max:     140
average: 6.79
median:  4

You would think that a simple XML serialization could be faster (almost 7 ms on average). To get some clue, let's profile the performance test with VisualVM :


Ouch, 95% of the execution time is taken by the JAXBContext method newInstance(). How to improve this ? A quick search on the internet shows that the JAXBContext is actually thread-safe, so it is not necessary to instantiate it each time.

The previous code can be improved:
public class UserSerializer {
    
    private static JAXBContext jaxbContext;
    
    static {
        try {
            jaxbContext = JAXBContext.newInstance(User.class);
        } catch (JAXBException ex) {
            // handles exception
            // ...
        }
    }

    public void serializeUserToXmlFile(User user, File file) throws JAXBException {
        Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
        jaxbMarshaller.marshal(user, file);
    }
}

Now, running the test again produces the output:

org.javabenchmark.UserSerializerTest.shouldSerializeRandomUser
samples: 200
max:     133
average: 1.4
median:  0


It is much better and then we can add performance checks to our test. To achieve this, ContiPerf provides the @Required annotation:
  • @Required(throughput = 20) : Requires to have at least 20 test executions per second 
  • @Required(average = 50): Requires an average execution time of not more than 50 milliseconds
  • @Required(median = 45): Requires that 50% of all executions do not take longer than 45 milliseconds
  • @Required(max = 2000): Requires that no invocation takes more than 2000 milliseconds (2 seconds)
  • @Required(totalTime = 5000): Requires that the sum of all execution times is not more than 5000 milliseconds (5 seconds)
  • @Required(percentile90 = 3000): Requires that 90% of all executions do not take longer than 3000 milliseconds
  • @Required(percentile95 = 5000): Requires that 95% of all executions do not take longer than 5000 milliseconds
  • @Required(percentile99 = 10000): Requires that 99% of all executions do not take longer than 10000 milliseconds
  • @Required(percentiles = "66:200,96:500"): Requires that 66% of all executions do not take longer than 200 milliseconds and 96% of all executions do not take longer than 500 milliseconds

In our case, we could check that the average does not exceed 2 ms and that 95% of all executions do not take longer that 1 ms:
public class UserSerializerTest {

    @Rule
    public ContiPerfRule i = new ContiPerfRule();

    @Test
    @PerfTest(invocations = 200, threads = 2)
    @Required(average = 2, percentile95 = 1)
    public void shouldSerializeRandomUser() throws JAXBException {
        // ...
    }
}

Conclusion
To enable continuous performance in your test:
  1. write a JUnit test that passes
  2. transform it into a performance one
  3. ensure that performance is fine
  4. add your performance requirements

8 comments:

  1. ContiPerf link is pointing to blogger.com

    ReplyDelete
    Replies
    1. You are right, my mistake. The link is now fixed. Thanks.

      Delete
  2. This looks a great tool, didn't know about it before. By the way, I have also shared few unit testing best practices, let me know how do you find it..

    ReplyDelete
    Replies
    1. I agree with you, the practices you describe are fondamental. To facilitate/automate point 3, i would suggest to give a look at http://databene.org/feed4junit.html

      Delete
    2. I forgot to add that i recommend to write your test controls/assertions with https://github.com/alexruiz/fest-assert-2.x

      Delete
  3. Hi, Good staff.
    If you know, I try to use contiperf with ant to generate csv report but I can't get it.
    Would you help to do it, please, if you can?

    A.L

    ReplyDelete
  4. If you like fest assert, give a try to AssertJ, it's a fork of fest assert which is not active anymore.

    Here it is : https://github.com/joel-costigliola/assertj-core

    Cheers,

    Joel

    ReplyDelete
    Replies
    1. Hi Joel, thanks for the tip, i will give it a try.

      Delete