Sunday, February 24, 2013

Benchmarking With JUnitBenchmark

If you are interested in doing some benchmark with JUnit, there is a very useful library for that: JUnitBenchmark. It has really nice features like:
  • Recording of execution time (average and standard deviation)
  • Monitoring garbage collector activity
  • Benchmark warm-up

To demonstrate what can be done with it, let's go with a trivial example comparing 2 components that handle String concatenation:
  1. StringBuilder from the JDK
  2. TextBuilder from Javolution

The first thing is to add dependencies for JUnitBenchmark and Javolution into the pom.xml:
        
        
        
        
            junit
            junit
            4.11
            test
        
        
        
        
            com.carrotsearch
            junit-benchmarks
            0.5.0
            test
        
        
        
        
            org.javolution
            javolution
            5.3.1
            test
        

Then here is the code for the JUnit Benchmark:
package org.javabenchmark;

import com.carrotsearch.junitbenchmarks.AbstractBenchmark;
import com.carrotsearch.junitbenchmarks.BenchmarkOptions;
import javolution.text.TextBuilder;
import org.junit.Test;

/**
 * Benchmark for String concatenation. Compares StringBUilder (JDK) and
 * TextBuilder (Javolution).
 */
public class StringConcatenationBenchmark extends AbstractBenchmark {

    public static final long LOOPS_COUNT = 10000000;

    @Test
    @BenchmarkOptions(benchmarkRounds = 3, warmupRounds = 1)
    public void stringBuilderBenchmark()  {
        
        StringBuilder builder = new StringBuilder();
        for (long i = 0; i < LOOPS_COUNT; i++) {
            builder.append('i').append(i);
        }
        System.out.println(builder.toString().length());
    }
    
    @Test
    @BenchmarkOptions(benchmarkRounds = 3, warmupRounds = 1)
    public void textBuilderBenchmark()  {
        
        TextBuilder builder = new TextBuilder();
        for (long i = 0; i < LOOPS_COUNT; i++) {
            builder.append('i').append(i);
        }
        System.out.println(builder.toString().length());
    }
}


And the output produced by the test on my machine (Ubuntu VM with 5Go and Core i7):

  • StringConcatenationBenchmark.stringBuilderBenchmark: [measured 3 out of 4 rounds, threads: 1 (sequential)] round: 0.68 [+- 0.06], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 6, GC.time: 0.19, time.total: 2.66, time.warmup: 0.64, time.bench: 2.03
  • StringConcatenationBenchmark.textBuilderBenchmark: [measured 3 out of 4 rounds, threads: 1 (sequential)] round: 0.47 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 5, GC.time: 0.18, time.total: 1.97, time.warmup: 0.57, time.bench: 1.41

Conclusion:
This micro-benchmark demonstrated that TextBuilder is 30% faster than StringBuilder (in our trivial test) and more stable (+- 0.01 compared to +- 0.06). GC metrics are similar in both cases.

2 comments:

  1. Typos in the POM example: it should be , not (note the capital 'I').

    Great example, though. :)

    ReplyDelete
    Replies
    1. I think that some XML parts of your message are missing :)

      Delete