OUT OF MEMORY ERROR

Let us continue our series of posts about solving the OutOfMemoryError in our hypothetical production system.

We have described different methods to tackle the problem, and today's post concentrates on what you can learn from heap dumps. Spoiler alert: with a bit of luck, you can get very close to solving the OOM :)

In retrospect, these are the methods we have already tried:

1. We have added more physical memory to that machine. Nothing changed.
   
2. We have attached memory profiler to it. Our application became ten times slower, and lived ten times longer.
3. We have wrestled with command line tools and saw some magic numbers.

None of them actually helped us solve the actual cause of the problem. Today we will try the next weapon in our arsenal: the memory dumps and the tools that help you work with them - the memory dump analyzers.

Just one more remark before we begin – note that some authors use the term “heap dump” for describing memory dumps. In the Java world, most of the time heap dump and memory dump mean the same thing. I in this post I will use them both interchangeably.

What is a "memory dump"?

Memory dump is a snapshot of Java Virtual Machine’s memory, taken at one specific moment. Usually it is saved on the disk for further analysis.

Using the dump, JVM memory contents can be investigated at developer’s leisure, using a wide range of tools, in the developer’s comfortable environment far away from the sensitive production site. The ultimate goal of that investigation is to find objects that consume too much memory and where those objects are being held in the running application.

The practical examples below will once more be based on our sample leaking PetClinic application, distributed with the Plumbr release package.

How can you create the memory dump?

The memory dump can be created in two ways:

1. By adding “-XX:+HeapDumpOnOutOfMemoryError” to your java start command, like this:java -XX:+HeapDumpOnOutOfMemoryError -Xmx512m ...When you start your JVM like this, whenever an OutOfMemoryError (OOM) is thrown by the JVM, a full memory dump will be written to the disk just before crashing/exiting/whatever your JVM does at an OOM.
   
   Although waiting for the JVM to crash before starting to look for the causes of the problem could be a little harsh, I strongly suggest that no JVM run in production without this parameter. Let me stress that again: Go and add that parameter in your production server configuration now! The reason is very straightforward: if your production server would ever suffer from an OutOfMemoryError, you will want to possess that memory dump. It will be the most useful data for postmortem analysis and often the dump alone will be sufficient to find out the cause of the crash.