Glassfish 3.1.2.2 Web Service Memory Leak

I recently installed Netbeans 7.3 with the Glassfish 3.1.2.2 combo, and decided to play with it. So i deployed a simple application that exposes an "echo" web service.

@WebService(serviceName = "EchoWebService")
public class EchoWebService {

    @WebMethod
    public String echo(@WebParam(name="text") String text) {
        return text;
    }
}

Then using JMeter, i stressed the service to see how the server performs: a simple test plan that consists of a group of 5 threads, each thread sending continuously a HTTP request to the web service. The request contains a SOAP envelope holding the parameter that must be echoed by the web service.

JMeter Sampler - HTTP Request with SOAP Envelope

This chart illustrates the performance monitored within JMeter (in request per second):

Echo Web Service - Requests per second

As you can see, after 2:24 mn the server starts performing very badly .. and after almost 5 minutes i stopped the load test.

During the test, i monitored the server's memory with JConsole:

JConsole - Heap Usage

The graph shows that objects created during the test are not freed by the garbage collector and at the end the JVM is spending most of the time in trying to free some memory.

At this point, i ran the test another time and forced a heap dump to got some clue about the memory leak.

The analysis reveals that the leak comes from the com.sun.enterprise.container.common.impl.managedbean.ManagedBeanManagerImpl component that holds an instance of a java.util.HashMap$Entry[] where 91% of the memory is accumulated:

Memory Analyzer Tool - Report

This array is full of org.glassfish.weld.services.JCDIServiceImpl$JCDIInjectionContextImpl instances (524 488 instances).

I do not know if it is a known issue, or if there is something wrong with my modus operandi.

Benchmarking With JUnitBenchmark

If you are interested in doing some benchmark with JUnit, there is a very useful library for that: JUnitBenchmark. It has really nice features like:

• Recording of execution time (average and standard deviation)
• Monitoring garbage collector activity
• Benchmark warm-up

To demonstrate what can be done with it, let's go with a trivial example comparing 2 components that handle String concatenation:

1. StringBuilder from the JDK
2. TextBuilder from Javolution

The first thing is to add dependencies for JUnitBenchmark and Javolution into the pom.xml:

        
        
        
        
            junit
            junit
            4.11
            test
        
        
        
        
            com.carrotsearch
            junit-benchmarks
            0.5.0
            test
        
        
        
        
            org.javolution
            javolution
            5.3.1
            test
        

Then here is the code for the JUnit Benchmark:

package org.javabenchmark;

import com.carrotsearch.junitbenchmarks.AbstractBenchmark;
import com.carrotsearch.junitbenchmarks.BenchmarkOptions;
import javolution.text.TextBuilder;
import org.junit.Test;

/**
 * Benchmark for String concatenation. Compares StringBUilder (JDK) and
 * TextBuilder (Javolution).
 */
public class StringConcatenationBenchmark extends AbstractBenchmark {

    public static final long LOOPS_COUNT = 10000000;

    @Test
    @BenchmarkOptions(benchmarkRounds = 3, warmupRounds = 1)
    public void stringBuilderBenchmark()  {
        
        StringBuilder builder = new StringBuilder();
        for (long i = 0; i < LOOPS_COUNT; i++) {
            builder.append('i').append(i);
        }
        System.out.println(builder.toString().length());
    }
    
    @Test
    @BenchmarkOptions(benchmarkRounds = 3, warmupRounds = 1)
    public void textBuilderBenchmark()  {
        
        TextBuilder builder = new TextBuilder();
        for (long i = 0; i < LOOPS_COUNT; i++) {
            builder.append('i').append(i);
        }
        System.out.println(builder.toString().length());
    }
}

And the output produced by the test on my machine (Ubuntu VM with 5Go and Core i7):

• StringConcatenationBenchmark.stringBuilderBenchmark: [measured 3 out of 4 rounds, threads: 1 (sequential)] round: 0.68 [+- 0.06], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 6, GC.time: 0.19, time.total: 2.66, time.warmup: 0.64, time.bench: 2.03

• StringConcatenationBenchmark.textBuilderBenchmark: [measured 3 out of 4 rounds, threads: 1 (sequential)] round: 0.47 [+- 0.01], round.block: 0.00 [+- 0.00], round.gc: 0.00 [+- 0.00], GC.calls: 5, GC.time: 0.18, time.total: 1.97, time.warmup: 0.57, time.bench: 1.41

Conclusion:
This micro-benchmark demonstrated that TextBuilder is 30% faster than StringBuilder (in our trivial test) and more stable (+- 0.01 compared to +- 0.06). GC metrics are similar in both cases.

Servlet Monitoring with Metrics

Metrics from Coda Hale is a wonderful library to ease the monitoring of the performance of your code. It gives you tons of metrics directly via JMX, logs or CSV files.

I use it to monitor the performance of servlet in a JEE6 application. It is very simple to set up:

1. Add the following dependency to your pom.xml:

           
           
               com.yammer.metrics
               metrics-core
               2.2.0
           
   

2. Declare a Timer:

       /**
        * the timer for monitoring incoming HTTP POST requests.
        */
       private final Timer postTimer = Metrics.newTimer(getClass(), "post-requests-timer", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
   

   This code instantiates a new Timer where duration is measured in milliseconds and rate in seconds (to compute request per second).


3. Use the previous Timer when handling HTTP request:

       /**
        * Handles the HTTP
        * POST method.
        *
        * @param request servlet request
        * @param response servlet response
        * @throws ServletException if a servlet-specific error occurs
        * @throws IOException if an I/O error occurs
        */
       @Override
       protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
           // starts stopwatch
           final TimerContext context = postTimer.time();
   
           // handles request and response, calls business logic
           ...
   
           // ends stopwatch
           context.stop();
       }
   

4. Put some load on your servlet and monitor it using JConsole for instance: