Debugging dynamic ClassLoaders in heap dumps
Dynamic class loading is very hard to do correctly. There are a lot of things which can go wrong. It is impossible to think about all eventualities while developing new features. Hence, developers need to take heap dumps of the JVM to analyze the runtime behavior and memory state of the system to find out about strongly held references to objects which should have been garbage collected. The Eclipse Memory Analyzer Tool (MAT) is a great way to find problems quickly. Without MAT, it would cost weeks of very detailed code review and reverse engineering.The first step in debugging dynamic class loading issues is to identify the ClassLoader of the system which are currently alive. MAT provides an object query language (OQL) to perform queries against the Eclipse Memory Analyzer Tool model of the heap dump. It is possible to query for objects, their attributes and their immediate dominators. In general, finding all ClassLoaders is a simple query:
Such a general query will result in thousands of object instances of type java.lang.ClassLoader and all its subtypes. The query can be made more specific by adding a WHERE clause to find only ClassLoaders of our own types, e.g.
This query result shows us that there are two types of ClassLoaders and each has many instances: AggregatedClassLoader and JarClassLoader. The AggregatedClassLoader contains a flat list of other ClassLoaders (not a hierarchy) to load classes from that ‘child’ ClassLoader where they are first found. The AggregatedClassLoader is used for Bean Mapping of Rule Model Structures – it maps a Visual Rrules datatype from one rule model to a similar VR datatype of another rule model (RuleModelBeansMappingService). Selecting one (or multiple) of the AggregatedClassLoaders and running the “Merge Shortest Path To GC Roots” operation, the Eclipse Memory Analyzer Tool shows a tree/path to the object holding a reference to these ClassLoaders. Look at the following object tree:
The AggregatedClassLoader object with object address 0x5343340 is kept in a WeakHashMap called “valueByClassLoader” in an anonymous class (BeanUtilsBean$1) within the BeanUtilsBean class. As the field is called “beansByClassLoader“, it seems the application uses commons-beanutils-1.7.0, because the field was renamed to “BEANS_BY_CLASSLOADER” in commons-beanutils-1.8.0. Following the source code, reading the JavaDoc of BeanUtils and googling for it will result in finding BEANUTILS-291, BEANUTILS-59, JBPAPP-713 and multiple references to memory leaks in the Change Log without mentioning JIRA issues. In the test cases of beanutils, their code explicitly clears caches and I wonder if our application has to do the same thing:
Also, reading the JavaDoc of ContextClassLoaderLocal, which is used by BeanUtilsBean, reveals some interesting notes:
This class takes some care to ensure that when a component which uses this class is “undeployed” by a container the component-specific classloader and all its associated classes (and their static variables) are garbage-collected. Unfortunately there is one scenario in which this does NOT work correctly and there is unfortunately no known workaround other than ensuring that the component (or its container) calls the “unset” method on this class for each instance of this class when the component is undeployed. The problem occurs if:
- the class containing a static instance of this class was loaded via a shared classloader, and
- the value stored in the instance is an object whose class was loaded via the component-specific classloader (or any of the objects it refers to were loaded via that classloader).
This sounds like a serious issue, esp. since most libraries assume that WebApps don’t do dynamic ClassLoading. But honestly, this is way too complicated to grasp without drawing a picture of the whole situation and without someone else to talk about it. After this longly detour in the depths of third-party libraries, let’s come get back to where we left with the OQL queries: We can add additional information to the result table by specifying more columns in the SELECT clause, such as the value of an attribute of an object within the found ClassLoader object and meta-data such as the retained heap size (the retained heap size is the amount of memory which could be freed by the garbage collector if this object could be garbage collected). In the following query, we ask for all JarClassLoader (these are the ones which dynamically load Visual Rules generated rule model code from the database, and show the name of the rule model and the persistent ID if the JAR archive in the database.
Eclipse Memory Analyzer Tool’s Object Query Language is a great way of analyzing the heap dump, to get an overview of live objects and their relationships. Each row in the OQL result table is a MAT Object and various operations can be performed upon it, such as finding all the objects involved in keeping the selected object alive (Merge Shortest Paths to GC Roots):
And if you have more complex OQL which MAT is unable to process, you can also try VisualVM, which has an even more sophisticated way to query the heap.