Thursday, March 7, 2013

Continuous Performance and JUnit with ContiPerf

If you want to introduce some continuous performance into your Java project, you should definitively check the ContiPerf library : that tool transforms your unit tests into performance tests simply by adding few annotations !

For instance, suppose your need to do some XML serialization with JAXB as illustrated below and therefore you have to test the UserSerializer class.


The corresponding code for the UserSerializer class is:
package org.javabenchmark;

import java.io.File;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;

public class UserSerializer {

    public void serializeUserToXmlFile(User user, File file) throws JAXBException {

        JAXBContext jaxbContext = JAXBContext.newInstance(User.class);
        Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
        jaxbMarshaller.marshal(user, file);
    }
}
and the corresponding code for the UserSerializer test would be:
package org.javabenchmark;

import java.io.File;
import java.util.UUID;
import javax.xml.bind.JAXBException;
import org.junit.Test;
import static org.fest.assertions.api.Assertions.*;

public class UserSerializerTest {
    
    @Test
    public void shouldSerializeRandomUser() throws JAXBException {
        
        // given a random user
        String id = UUID.randomUUID().toString();
        String firstName = "Julien";
        String lastName = "Paoletti";
        
        User randomUser = new User();
        randomUser.setId(id);
        randomUser.setFirstName(firstName);
        randomUser.setLastName(lastName);
        
        // when serializing it to a random XML file
        File xmlFile = new File("target/" + id + ".xml");
        xmlFile.deleteOnExit();
        
        UserSerializer serializer = new UserSerializer();
        serializer.serializeUserToXmlFile(randomUser, xmlFile);
        
        // then the file contains the corresponding XML
        String validXmlContent = ""
                + ""+ firstName + ""
                + ""+ lastName + ""
                + "";
        assertThat(xmlFile).hasContent(validXmlContent);
    }
}

The test passes,  it is now time to transform this unit test into a performance one in 2 steps:
  1. add a JUnit rule from ContiPerf
  2. add a ContiPerf annotation to configure the execution of the performance test

public class UserSerializerTest {

    @Rule
    public ContiPerfRule i = new ContiPerfRule();
    
    @Test
    @PerfTest(invocations = 200, threads = 2)
    public void shouldSerializeRandomUser() throws JAXBException {

        // ...
    }
}
As you can see, the test will be exectuted 200 times by 2 threads. On my computer (Ubuntu VM 5Go 4 CPU), running the test produces the following output:

org.javabenchmark.UserSerializerTest.shouldSerializeRandomUser
samples: 200
max:     140
average: 6.79
median:  4

You would think that a simple XML serialization could be faster (almost 7 ms on average). To get some clue, let's profile the performance test with VisualVM :


Ouch, 95% of the execution time is taken by the JAXBContext method newInstance(). How to improve this ? A quick search on the internet shows that the JAXBContext is actually thread-safe, so it is not necessary to instantiate it each time.

The previous code can be improved:
public class UserSerializer {
    
    private static JAXBContext jaxbContext;
    
    static {
        try {
            jaxbContext = JAXBContext.newInstance(User.class);
        } catch (JAXBException ex) {
            // handles exception
            // ...
        }
    }

    public void serializeUserToXmlFile(User user, File file) throws JAXBException {
        Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
        jaxbMarshaller.marshal(user, file);
    }
}

Now, running the test again produces the output:

org.javabenchmark.UserSerializerTest.shouldSerializeRandomUser
samples: 200
max:     133
average: 1.4
median:  0


It is much better and then we can add performance checks to our test. To achieve this, ContiPerf provides the @Required annotation:
  • @Required(throughput = 20) : Requires to have at least 20 test executions per second 
  • @Required(average = 50): Requires an average execution time of not more than 50 milliseconds
  • @Required(median = 45): Requires that 50% of all executions do not take longer than 45 milliseconds
  • @Required(max = 2000): Requires that no invocation takes more than 2000 milliseconds (2 seconds)
  • @Required(totalTime = 5000): Requires that the sum of all execution times is not more than 5000 milliseconds (5 seconds)
  • @Required(percentile90 = 3000): Requires that 90% of all executions do not take longer than 3000 milliseconds
  • @Required(percentile95 = 5000): Requires that 95% of all executions do not take longer than 5000 milliseconds
  • @Required(percentile99 = 10000): Requires that 99% of all executions do not take longer than 10000 milliseconds
  • @Required(percentiles = "66:200,96:500"): Requires that 66% of all executions do not take longer than 200 milliseconds and 96% of all executions do not take longer than 500 milliseconds

In our case, we could check that the average does not exceed 2 ms and that 95% of all executions do not take longer that 1 ms:
public class UserSerializerTest {

    @Rule
    public ContiPerfRule i = new ContiPerfRule();

    @Test
    @PerfTest(invocations = 200, threads = 2)
    @Required(average = 2, percentile95 = 1)
    public void shouldSerializeRandomUser() throws JAXBException {
        // ...
    }
}

Conclusion
To enable continuous performance in your test:
  1. write a JUnit test that passes
  2. transform it into a performance one
  3. ensure that performance is fine
  4. add your performance requirements

37 comments:

  1. ContiPerf link is pointing to blogger.com

    ReplyDelete
    Replies
    1. You are right, my mistake. The link is now fixed. Thanks.

      Delete
  2. This looks a great tool, didn't know about it before. By the way, I have also shared few unit testing best practices, let me know how do you find it..

    ReplyDelete
    Replies
    1. I agree with you, the practices you describe are fondamental. To facilitate/automate point 3, i would suggest to give a look at http://databene.org/feed4junit.html

      Delete
    2. I forgot to add that i recommend to write your test controls/assertions with https://github.com/alexruiz/fest-assert-2.x

      Delete
  3. Hi, Good staff.
    If you know, I try to use contiperf with ant to generate csv report but I can't get it.
    Would you help to do it, please, if you can?

    A.L

    ReplyDelete
  4. If you like fest assert, give a try to AssertJ, it's a fork of fest assert which is not active anymore.

    Here it is : https://github.com/joel-costigliola/assertj-core

    Cheers,

    Joel

    ReplyDelete
    Replies
    1. Hi Joel, thanks for the tip, i will give it a try.

      Delete
  5. Hi, seems to be great tool.
    Do you know if it is possible to integrate contiperf with CI to track/ display trends with avarage/maximum performance values (not only relay on single report)?

    ReplyDelete
  6. Hi ,Thanks for this tool. very useful. I would like to change the report file name every time.instead of overwritting an existing file. can anyone help me on this ?

    ReplyDelete
  7. Hi !

    When using Contiperf it produces out to (in our case ) stdout. Iwould like to remove that output. Does there exist a way to do that?

    Currenltly we get
    KeyLoggerTest.loggingLogHighThreadTest
    samples: 18404212
    max: 32
    average: 0.0010268844979616622
    median: 0

    but this is actually not interesting. All I want to know is if the test failed or not.

    ReplyDelete
  8. Thanks for sharing such informative article on Loadrunner Automation testing tool. This load testing tool will provide most precise information about the quality of software. Loadrunner training in Chennai

    ReplyDelete
  9. Very nice, i like the way you explained. I also wrote something on similar lines on Automated Testing of Java apps. Hope you would like it - http://bit.ly/1M47DeI

    ReplyDelete
  10. Hi, how to integrate contiPerf reports to jenkins ??? it's possible to save them on db ???

    ReplyDelete
  11. Thanks for sharing this unique and informative content which provided me the required information.
    oracle training in chennai

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. This blog is so nice to me. I will continue to come here again and again. Visit my link as well. Good luck
    http://www.jualobataborsiherbal.com/ obat aborsi
    http://caramenggugurkankandungan.info/ cara menggugurkan kandungan
    http://obataborsi59.com/ obat aborsi
    http://obataborsi59.com/cara-menggugurkan-kandungan-dengan-cepat-dan-aman/ cara menggugurkan kandungan
    http://obattelatdatangbulan.info/ obat telat datang bulan
    http://klinikobataborsi.com/ jual obat aborsi
    http://jualobatpenggugurkandungan.net/ obat penggugur kandungan
    http://tandatandakehamilan.net/ tanda tanda kehamilan
    http://tandatandakehamilan.net/cara-cepat-dan-selamat-menggugurkan-kandungan/ cara menggugurkan kandungan

    ReplyDelete
  15. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharng this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete

  16. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..
    Android App Development Company

    ReplyDelete
  17. Nice information my sincere thanks for sharing this post and please continue to share this kind of post
    Software Testing Training in Chennai

    ReplyDelete
  18. The blog gave me idea to perform regression test on database My sincere Thanks for sharing this post and please continue to share this post before i read this blog i didn't have any knowledge about this but now i got some knowledge.
    softwaretesting training in Chennai

    ReplyDelete
  19. Really very nice blog information for this one and more technical skills are improve,i like that kind of post. Software Testing Training in Chennai | Selenium Training in Chennai

    ReplyDelete

  20. Nice it seems to be good post... It will get readers engagement on the article since readers engagement plays an vital role in every blog.i am expecting more updated posts from your hands.
    iOS App Development Company
    iOS App Development Company

    ReplyDelete
  21. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    web design Company
    web development Company
    web design Company in chennai
    web development Company in chennai
    web design Company in India
    web development Company in India

    ReplyDelete
  22. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Fitness SMS
    Fitness Text
    Salon SMS
    Salon Text
    Investor Relation SMS
    Investor Relation Text

    ReplyDelete
  23. it is really amazing...thanks for sharing....provide more useful information...
    Mobile app development company

    ReplyDelete
  24. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    web design Company
    web development Company
    web design Company in chennai
    web development Company in chennai
    web design Company in India
    web development Company in India

    ReplyDelete
  25. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Texting API
    Text message marketing
    Digital Mobile Marketing
    Sms API
    Sms marketing

    ReplyDelete