Ccna final exam - java, php, javascript, ios, cshap all in one. This is a collaboratively edited question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
Wednesday, May 23, 2012
Creating a memory leak with Java
I just had an interview, and I was asked to create a memory leak with Java. Needless to say I felt pretty dumb having no clue on how to even start creating one.
Here's a good way to create a true memory leak (objects inaccessible by running code but still stored in memory) in pure Java:
The application creates a long-running thread (or use a thread pool to leak even faster). The thread loads a class via an (optionally custom) ClassLoader. The class allocates a large chunk of memory (e.g. new byte[1000000]), stores a strong reference to it in a static field, and then stores a reference to itself in a ThreadLocal. Allocating the extra memory is optional (leaking the Class instance is enough), but it will make the leak work that much faster. The thread clears all references to the custom class or the ClassLoader it was loaded from. Repeat.
This works because the ThreadLocal keeps a reference to the object, which keeps a reference to its Class, which in turn keeps a reference to its ClassLoader. The ClassLoader, in turn, keeps a reference to all the Classes it has loaded. It gets worse because in many JVM implementations Classes and ClassLoaders are allocated straight into permgen and are never GC'd at all.
A variation on this pattern is why application containers (like Tomcat) can leak memory like sieve if you frequently redeploy applications that happen to use ThreadLocals in any way. (Since the application container uses Threads as described, and each time you redeploy the application a new ClassLoader is used.)
Static field holding object reference [esp final field]
class MemorableClass { static final ArrayList list = new ArrayList(100); }
Calling String.intern() on lengthy String
String str=readString(); // read lengthy string any source db,textbox/jsp etc.. // This will place the string in memory pool from which you cant remove str.intern();
A simple thing to do is to use a HashSet with an incorrect (or non-existent) hashCode() or equals(), and then keep adding "duplicates". Instead of ignoring duplicates as it should, the set will only ever grow and you won't be able to remove them.
If you want these bad keys/elements to hang around you can use a static field like
class BadKey { // no hashCode or equals(); public final String key; public BadKey(String key) { this.key = key; } }
Map map = System.getProperties(); map.put(new BadKey("key"), "value"); // Memory leak even if your threads die.
Below there will be non-obvious case where java leaks, besides the standard case of forgotten listeners, static references, bogus/modifiable keys in hashmaps, or just threads stuck w/o any chance to end their life-cycle.
File.deleteOnExit() - always leaks the string, if the string is a substring the leak is even worse (the underlying char[] is also leaked), @Daniel, no needs for votes, though.
I'll concentrate on Threads to show the danger of unmanaged threads mostly, don't wish to even touch swing.
Runtime.addShutdownHook and not remove... and then even w/ removeShutdownHook due to a bug in ThreadGroup class regarding unstarted Threads it may not get collected, effectively leak the ThreadGroup. JGroup has the leak in GossipRouter. Creating but not starting a Thread goes into the same category as above. Creating a thread inherits the ContextClassLoader and AccessControlContext, plus the ThreadGroup and any InheritedThreadLocal, all those references are potential leaks, along w/ the entire classes loaded by the classloader and all static references, and ja-ja. The effect is especially visible w/ the entire j.u.c.Executor framework that features a super simple ThreadFactory interface, yet most developers have no clue of the lurking danger. Also a lot of libraries do start threads upon request (edit: way too many industry popular libraries) ThreadLocal caches, those are evil in many cases. I am sure everyone has seen quite a bit of simple caches based on ThreadLocal, well the bad news: if the thread keeps going more than expected life the context ClassLoader, it is a pure nice little leak. Do not use ThreadLocal caches unless really needed. Calling ThreadGroup.destroy() when the ThreadGroup has no Threads itself but still keeps child ThreadGroups. A bad leak that will prevent the ThreadGroup to remove from its parent but all the children become un-enumerateable. Using WeakHashMap and the value (in)directly references the key, this is a hard one to find w/o a heap dump. That applies to all extended Weak/SoftReference that might keep a hard reference back to the guarded object. Using java.net.URL w/ http(s) protocol and loading the resource from(!). This one is special, the KeepAliveCache creates a new thread in the system ThreadGroup which leaks the current thread's context classloader. The thread is created upon the first request when no alive thread exists, so either you may get lucky or just leak. There are few more cases (like ImageFetcher) of creating similar threads. Using InflaterInputStream passing new java.util.zip.Inflater() in the c-tor (PNGImageDecoder for instance) and not calling end() of the inflater. Well, if you pass in the c-tor w/ just new, no chance... and yes calling close() on the stream does not close the inflater if it's manually passed as c-tor parameter. This is not a true leak since it'd be released by the finalizer... when it deems it necessary. Till that moment it eats native memory so badly it can cause linux oom_killer to kill the process with impunity. The main issue is that finalization in java is very unreliable and G1 made it worse till 7.0.2. Moral of the story: release native resources as soon as you can, the finalizer is just too poor. Same case w/ java.util.zip.Deflater, this one is far worse since Deflater is memory hungry in java, i.e. always uses 15bits (max) and 8memory level (9 is max) allocating several hundreds KB of native mem. Fortunately, Deflater is not widely used and to my knowledge JDK contains no misuses. Always call end() if you manually create a Deflater or Inflater. The best part of the last two: you can't find them via normal profiling tools available.
(I can add some more time wasters I have encountered upon request)
The following is a pretty pointless example, if you do not understand JDBC. Or atleast how JDBC expects a developer to close Connection, Statement and ResultSet instances before discarding them or losing references to them, instead of relying on the implementation of finalize.
void doWork() { try { Connection conn = ConnectionFactory.getConnection(); PreparedStatement stmt = conn.preparedStatement("some query"); // executes a valid query ResultSet rs = stmt.executeQuery(); while(rs.hasNext()) { ... process the result set } } catch(SQLException sqlEx) { log(sqlEx); } }
The problem with the above is that the Connection object is not closed, and hence the physical connection will remain open, until the garbage collector comes around and sees that it is unreachable. GC will invoke the finalize method, but there are JDBC drivers that do not implement the finalize, at least not in the same way that Connection.close is implemented. The resulting behavior is that while memory will be reclaimed due to unreachable objects being collected, resources (including memory) associated with the Connection object might simply not be reclaimed.
In such an event where the Connection's finalize method does not clean up everything, one might actually find that the physical connection to the database server will last several garbage collection cycles, until the database server eventually figures out that the connection is not alive (if it does), and should be closed.
Even if the JDBC driver were to implement finalize, it is possible for exceptions to be thrown during finalization. The resulting behavior is that any memory associated with the now "dormant" object will not be reclaimed, as finalize is guaranteed to be invoked only once.
The above scenario of encountering exceptions during object finalization is related to another other scenario that could possibly lead to a memory leak - Object resurrection. Object resurrection is often done intentionally by creating a strong reference to the object from being finalized, from another object. When object resurrection is misused it will lead to a memory leak in combination with other sources of memory leaks.
There are plenty more examples that you can conjure up - like
managing a List instance where you are only adding to the list and not deleting from it (although you should be getting rid of elements you no longer need), or opening Sockets or Files but not closing them when they are no longer needed (similar to the above example involving the Connection class). not unloading Singletons when bringing down a Java EE application. Apparently, the Classloader that loaded the Singleton class will retain a reference to the class, and hence the singleton instance will never be collected. When a new instance of the application is deployed, a new class loader is usually created, and the former class loader will continue to exist due to the singleton.
The answer depends entirely on what the interviewer thought they were asking.
Is it possible in practice to make Java leak? Of course it is, and there are plenty of examples in the other answers.
But there are multiple meta-questions that may have been being asked?
Is a theoretically "perfect" Java implementation vulnerable to leaks? Does the candidate understand the difference between theory and reality? Does the candidate understand how garbage colleciton works? Or how garbage collection is supposed to work in an ideal case? Do they know they can call other languages through native interfaces? Do they know to leak memory in those other languages? Does the candidate even know what memory management is, and what is going on behind the scese in Java?
I'm reading your meta-question as "What's an answer I could have used in this interview situation". And hence, I'm going to focus on interview skills instead of Java. I believe your more likely to repeat the situaiton of not knowing the answer to a question in an interview than you are to be in a place of needing to know how to make Java leak. So, hopefully, this will help.
One of the most important skills you can develop for interviewing is learning to activley listen to the questions and working with the interviewer to extract their intent. Not only does this let you answer their question the way they want, but also shows that you have some vital communication skills. And when it comes down to a choice between many equally talented developers, I'll hire the one who listens, thinks, and understands before they respond every time.
Any time you keep references around to objects that you no longer need you have a memory leak. See Handling memory leaks in Java programs for examples of how memory leaks manifest themselves in Java and what you can do about it.
Most examples here are "too complex". They are edge case. With theses examples, the programmer made a mystake (like don't redifining equals/hashcode), or has been biten by a corner case of the JVM/JAVA (load of class with static...). I think that's not the type of example an interviewer want or even the most common case.
But there are really simpler case to memory leaks. Garbage collector only free what is no longer referenced. We as Java developper don't care about memory. We allocate it when needed and let it being freed automatically. Fine.
But any long lived application tend to have shared state. It can be anything, statics, singletons... Often non trivial application tend to make complex objects graphs. Just forgetting to set a reference to null or more often forgetting to remove one object from a collection is enough to make a memory leak.
Of course all sort of listeners (like UI listeners), caches, or any long lived shared state tend to produce memory leak if not properluy handled. What shall be understood is that this is not a Java corner case, or a problem with the garbage collector. It a design problem. We design that we add listener to a long lived object, but we don't remove the listener when no longer needed. We cache objects, but we have no strategy to remove them from the cache.
We maybe have a complex graph that store previous state that is needed by computation. But the previous state is itself linked to the state before and so on.
Like we have to close SQL connections or files. We need to set proper referenres to null and remove elements from collection. We shall have proper caching strategies (maximum memory size, number of elements, or timers). All object that allow listener to be notified must provide both a addListener and removeListener method. And when theses notifier are no longer used, they must clear their listener list.
MemoryLeak is indeed trully possible and is perfectly predictible. No need for special language features or corner case. Memory leaks are either an indicator that something is maybe missing or even of design problems.
Probably one of the simplest examples is the implementation of ArrayList.remove(int):
public E remove(int index) { RangeCheck(index);
modCount++; E oldValue = (E) elementData[index];
int numMoved = size - index - 1; if (numMoved > 0) System.arraycopy(elementData, index + 1, elementData, index, numMoved); elementData[--size] = null; // Let gc do its work
return oldValue; }
Would you have thought to clear the array element that is no longer used? That reference might keep a huge object alive ...
I can copy my answer from here: Easiest way to cause memory leak in Java?
"A memory leak, in computer science (or leakage, in this context), occurs when a computer program consumes memory but is unable to release it back to the operating system." (Wikipedia)
The easy answer is: You can't. Java does automatic memory management and will free resources that are not needed for you. You can't stop this from happening. It will ALWAYS be able to release the resources. In programs with manual memory management, this is different. You cann get some memory in C using malloc(). To free the memory, you need the pointer that malloc returned and call free() on it. But if you don't have the pointer anymore (overwritten, or lifetime exceeded), then you are unfortunately incapable of freeing this memory and thus you have a memory leak.
All the other answers so far are in my definition not really memory leaks. They all aim at filling the memory with pointless stuff real fast. But at any time you could still dereference the objects you created and thus freeing the memory --> NO LEAK. acconrad's answer comes pretty close though as I have to admit since his solution is effectively to just "crash" the garbage collector by forcing it in an endless loop).
The long answer is: You can get a memory leak by writing a library for Java using the JNI, which can have manual memory management and thus have memory leaks. If you call this library, your java process will leak memory. Or, you can have bugs in the JVM, so that the JVM looses memory. There are probably bugs in the JVM, there may even be some known ones since garbage collection is not that trivial, but then it's still a bug. By design this is not possible. You may be asking for some java code that is effected by such a bug. Sorry I don't know one and it might well not be a bug anymore in the next Java version anyway.
Take any web application running in any servlet container (Tomcat, Jetty, Glassfish, whatever...). Redeploy the app 10 or 20 times in a row (it may be enough to simply touch the WAR in the server's autodeploy directory.
Unless anybody has actually tested this, chances are high that you'll get an OutOfMemoryError after a couple of redeployments, because the application did not take care to clean up after itself. You may even find a bug in your server with this test.
The problem is, the lifetime of the container is longer than the lifetime of your application. You have to make sure that all references the container might have to objects or classes of your application can be garbage collected.
If there is just one reference surviving the undeployment of your web app, the corresponding classloader and by consequence all classes of your web app cannot be garbage collected.
Threads started by your application, ThreadLocal variables, logging appenders are some of the usual suspects to cause classloader leaks.
You able to make memory leak with sun.misc.Unsafe class. In fact this service class is used in different standart classes (for example in java.nio classes). You can't create instance of this class directly, but you may use reflection to do that.
Code doesn't compile in Eclipse IDE - compile it using command javac (during compilation you'll get warnings)
Here's a simple/sinister one via http://wiki.eclipse.org/Performance_Bloopers#String.substring.28.29.
public class StringLeaker { private final String muchSmallerString;
public StringLeaker() { // Imagine the whole Declaration of Independence here String veryLongString = "We hold these truths to be self-evident...";
// The substring here maintains a reference to the internal char[] // representation of the original string. this.muchSmallerString = veryLongString.substring(0, 1); } }
Because the substring refers to the internal representation of the original, much longer string, the original stays in memory. Thus, as long as you have a StringLeaker in play, you have the whole original string in memory, too, even though you might think you're just holding on to a single-character string.
The way to avoid storing an unwanted reference to the original string is to do something like this:
... this.muchSmallerString = new String(veryLongString.substring(0, 1)); ...
For added badness, you might also .intern() the substring:
But that is about a "standard" type of memory leak, when you cannot access the memory anymore, but it is still owned by the application. You can instead keep references to unused objects, or open streams without closing them afterwards.
A common example of this in GUI code is when creating a widget/component and adding a listener to some static/application scoped object and then not removing the listener when the widget is destroyed. Not only do you get a memory leak but a performance hit as when whatever you are listening too fires events all your old listeners are called too.
I have had a nice "memory leak" in relation to PermGen and xml-parsing once. The XMl-Parser we used (cant remember which one it was) did a String.intern() on tag names, to make comparison faster. One of our customers had the gread idea to store data values not in xml-attributes or text, but as tagnames, so we had a document like:
<data> <1>bla</1> <2>foo</> ... </data>
In fact, they did not use numbers but longer textual id's (around 20chars), which where unique and came in at a rate of 10-15 Million a day. That makes 200MB of rubbish a day, which is never needed again, and never GCed (since it is in permgen). We had permgen set to 512MB, so it took around 2 days for the OOME to arrive...
I thought it was interesting that no one used the internal class examples. If you have an internal class; it inherently maintains a reference to the containing class. Of course it is not technically a memory leak because Java WILL eventually clean it up; but this can cause classes to hang around longer than anticipated.
public class Example1 { public Example2 getNewExample2() { return this.new Example2(); } public class Example2 { public Example2() {} } }
Now if you call Example1 and get an Example2 discarding Example1, you will inherently still have a link to an Example1 object.
public class Referencer { public static Example2 GetAnExample2() { Example1 ex = new Example1(); return ex.getNewExample2(); }
public static void main(String[] args) { Example2 ex = Referencer.GetAnExample2(); // As long as ex is reachable; Example1 will always remain in memory. } }
I've also heard a rumor that if you have a variable that exists for longer than a specific amount of time; Java assumes that it will always exist and will actually never try to clean it up if cannot be reached in code anymore. But that is completely unverified.
I don't think anyone has said this yet: you can resurrect an object by overriding the finalize() method such that finalize() stores a reference of this somewhere. The garbage collector will only be called once on the object so after that the object will never destroyed.
The interviewer might have be looking for a circular reference solution:
public static void main(String[] args) { while (true) { Element first = new Element(); first.next = new Element(); first.next.next = first; } }
This is a classic problem with reference counting garbage collectors. You would then politely explain that JVMs use a much more sophisticated algorithm that doesn't have this limitation.
As a lot of people have suggested, Resource Leaks are fairly easy to cause - like the JDBC examples. Actual Memory leaks are a bit harder - especially if you aren't relying on broken bits of the JVM to do it for you...
The ideas of creating objects that have a very large footprint and then not being able to access them aren't real memory leaks either. If nothing can access it then it will be garbage collected, and if something can access it then it's not a leak...
One way that used to work though - and I don't know if it still does - is to have a three-deep circular chain. As in Object A has a reference to Object B, Object B has a reference to Object C and Object C has a reference to Object A. The GC was clever enough to know that a two deep chain - as in A <--> B - can safely be collected if A and B aren't accessible by anything else, but couldn't handle the three-way chain...
I think that a valid example could be using ThreadLocal variables in an environment where threads are pooled.
For instance, using ThreadLocal variables in Servlets to communicate with other web components, having the threads being created by the container and maintaining the idle ones in a pool. ThreadLocal variables, if not correctly cleaned up, will live there until, possibly, the same web component overwrites their values.
Of course, once identified, the problem can be solved easily.
I recently encountered a memory leak situation caused in a way by log4j.
Log4j has this mechanism called Nested Diagnostic Context(NDC) which is is an instrument to distinguish interleaved log output from different sources. The granularity at which NDC works is threads, so it distinguishes log outputs by different threads separately.
In order to store thread specific tags, log4j's NDC class uses a Hashtable which is keyed by the Thread object itself (as opposed to say the thread id), and thus till the NDC tag stays in memory all the objects that hang off of the thread object also stay in memory. In our web application we use NDC to tag logoutputs with a request id to distinguish logs from a single request separately. The container that associates the NDC tag with a thread, also removes it while returning the response from a request. The problem occured when during the course of processing a request, a child thread was spawned, something like the following code:
pubclic class RequestProcessor { private static final Logger logger = Logger.getLogger(RequestProcessor.class); public void doSomething() { .... final List<String> hugeList = new ArrayList<String>(10000); new Thread() { public void run() { logger.info("Child thread spawned") for(String s:hugeList) { .... } } }.start(); } }
So an NDC context was associated with inline thread that was spawned. The thread object that was the key for this NDC context, is the inline thread which has the hugeList object hanging off of it. Hence even after the thread finished doing what it was doing, the reference to the hugeList was kept alive by the NDC context Hastable, thus causing a memory leak.
there are many different situations memory will leak. One i encountered, which expose a map that should not be exposed and used in other place.
public class ServiceFactory {
private Map<String, Service> services;
private static ServiceFactory singleton;
private ServiceFactory() { services = new HashMap<String, Service>(); }
public static synchronized ServiceFactory getDefault() {
if (singleton == null) { singleton = new ServiceFactory(); } return singleton; }
public void addService(String name, Service serv) { services.put(name, serv); }
public void removeService(String name) { services.remove(name); }
public Service getService(String name, Service serv) { return services.get(name); }
// the problematic api, which expose the map. //and user can do quite a lot of thing from this api. //for example, create service reference and forget to dispose or set it null //in all this is a dangerous api, and should not expose public Map<String, Service> getAllServices() { return services; }
}
// resource class is a heavy class class Service {
An example I recently fixed is creating new GC and Image objects, but forgetting to call dispose() method.
GC javadoc snippet:
Application code must explicitly invoke the GC.dispose() method to release the operating system resources managed by each instance when those instances are no longer required. This is particularly important on Windows95 and Windows98 where the operating system has a limited number of device contexts available.
Image javadoc snippet:
Application code must explicitly invoke the Image.dispose() method to release the operating system resources managed by each instance when those instances are no longer required.
One possibility is to create a wrapper for an ArrayList that only provides one method: one that adds things to the ArrayList. Make the ArrayList itself private. Now, construct one of these wrapper objects in global scope (as a static object in a class) and qualify it with the final keyword (e.g. public static final ArrayListWrapper wrapperClass = new ArrayListWrapper()). So now the reference cannot be altered. That is, wrapperClass = null won't work and can't be used to free the memory. But there's also no way to do anything with wrapperClass other than add objects to it. Therefore, any objects you do add to wrapperClass are impossible to recycle.
use commons-logging in a servlet container (a bit provocative perhaps) start a thread in a servlet container and don't return from it's run method load animated gifs in a servlet container (this will start an animation thread)
The above effects could be 'improved' by redeploying the application ;)
Recently stumbled upon this:
Calling "new java.util.zip.Inflater();" without calling "Inflater.end()" ever
Read http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5072161 and linked issues for an in-depth-discussion.
You can create a moving memory leak by creating a new instance of a class in that class's finalize method. Bonus points if the finalizer creates multiple instances. Here's a simple program that leaks the entire heap in sometime between a few seconds and a few minutes depending on your heap size:
class Leakee { public void check() { if (depth > 2) { Leaker.done(); } } private int depth; public Leakee(int d) { depth = d; } protected void finalize() { new Leakee(depth + 1).check(); new Leakee(depth + 1).check(); } }
public class Leaker { private static boolean makeMore = true; public static void done() { makeMore = false; } public static void main(String[] args) throws InterruptedException { // make a bunch of them until the garbage collector gets active while (makeMore) { new Leakee(0).check(); } // sit back and watch the finalizers chew through memory while (true) { Thread.sleep(1000); System.out.println("memory=" + Runtime.getRuntime().freeMemory() + " / " + Runtime.getRuntime().totalMemory()); } } }
Here's a good way to create a true memory leak (objects inaccessible by running code but still stored in memory) in pure Java:
ReplyDeleteThe application creates a long-running thread (or use a thread pool to leak even faster).
The thread loads a class via an (optionally custom) ClassLoader.
The class allocates a large chunk of memory (e.g. new byte[1000000]), stores a strong reference to it in a static field, and then stores a reference to itself in a ThreadLocal. Allocating the extra memory is optional (leaking the Class instance is enough), but it will make the leak work that much faster.
The thread clears all references to the custom class or the ClassLoader it was loaded from.
Repeat.
This works because the ThreadLocal keeps a reference to the object, which keeps a reference to its Class, which in turn keeps a reference to its ClassLoader. The ClassLoader, in turn, keeps a reference to all the Classes it has loaded. It gets worse because in many JVM implementations Classes and ClassLoaders are allocated straight into permgen and are never GC'd at all.
A variation on this pattern is why application containers (like Tomcat) can leak memory like sieve if you frequently redeploy applications that happen to use ThreadLocals in any way. (Since the application container uses Threads as described, and each time you redeploy the application a new ClassLoader is used.)
Static field holding object reference [esp final field]
ReplyDeleteclass MemorableClass {
static final ArrayList list = new ArrayList(100);
}
Calling String.intern() on lengthy String
String str=readString(); // read lengthy string any source db,textbox/jsp etc..
// This will place the string in memory pool from which you cant remove
str.intern();
(Unclosed) open streams ( file , network etc... )
try {
BufferedReader br = new BufferedReader(new FileReader(inputFile));
...
...
} catch (Exception e) {
e.printStacktrace();
}
Unclosed connections
try {
Connection conn = ConnectionFactory.getConnection();
...
...
} catch (Exception e) {
e.printStacktrace();
}
Areas that are unreachable from JVM's garbage collector
like memory allocated through native methods
In web applications objects stored in application scope till application is restarted or removed explicitly
getServletContext().setAttribute("SOME_MAP", map);
In web applications objects stored session scope till its invalidated or removed explicitly
session.setAttribute("SOME_MAP", map);
Incorrect or inappropriate JVM options
like noclassgc option on IBM JDK that prevents unused class garbage collection
see IBM jdk settings
Some memory overheads
Using new Boolean() instead of Boolean.TRUE or Boolean.valueOf()
Boolean b = new Boolean(true);
Using new String()
String s = new String("why new?");
A simple thing to do is to use a HashSet with an incorrect (or non-existent) hashCode() or equals(), and then keep adding "duplicates". Instead of ignoring duplicates as it should, the set will only ever grow and you won't be able to remove them.
ReplyDeleteIf you want these bad keys/elements to hang around you can use a static field like
class BadKey {
// no hashCode or equals();
public final String key;
public BadKey(String key) { this.key = key; }
}
Map map = System.getProperties();
map.put(new BadKey("key"), "value"); // Memory leak even if your threads die.
Below there will be non-obvious case where java leaks, besides the standard case of forgotten listeners, static references, bogus/modifiable keys in hashmaps, or just threads stuck w/o any chance to end their life-cycle.
ReplyDeleteFile.deleteOnExit() - always leaks the string, if the string is a substring the leak is even worse (the underlying char[] is also leaked), @Daniel, no needs for votes, though.
I'll concentrate on Threads to show the danger of unmanaged threads mostly, don't wish to even touch swing.
Runtime.addShutdownHook and not remove... and then even w/ removeShutdownHook due to a bug in ThreadGroup class regarding unstarted Threads it may not get collected, effectively leak the ThreadGroup. JGroup has the leak in GossipRouter.
Creating but not starting a Thread goes into the same category as above.
Creating a thread inherits the ContextClassLoader and AccessControlContext, plus the ThreadGroup and any InheritedThreadLocal, all those references are potential leaks, along w/ the entire classes loaded by the classloader and all static references, and ja-ja. The effect is especially visible w/ the entire j.u.c.Executor framework that features a super simple ThreadFactory interface, yet most developers have no clue of the lurking danger. Also a lot of libraries do start threads upon request (edit: way too many industry popular libraries)
ThreadLocal caches, those are evil in many cases. I am sure everyone has seen quite a bit of simple caches based on ThreadLocal, well the bad news: if the thread keeps going more than expected life the context ClassLoader, it is a pure nice little leak. Do not use ThreadLocal caches unless really needed.
Calling ThreadGroup.destroy() when the ThreadGroup has no Threads itself but still keeps child ThreadGroups. A bad leak that will prevent the ThreadGroup to remove from its parent but all the children become un-enumerateable.
Using WeakHashMap and the value (in)directly references the key, this is a hard one to find w/o a heap dump. That applies to all extended Weak/SoftReference that might keep a hard reference back to the guarded object.
Using java.net.URL w/ http(s) protocol and loading the resource from(!). This one is special, the KeepAliveCache creates a new thread in the system ThreadGroup which leaks the current thread's context classloader. The thread is created upon the first request when no alive thread exists, so either you may get lucky or just leak. There are few more cases (like ImageFetcher) of creating similar threads.
Using InflaterInputStream passing new java.util.zip.Inflater() in the c-tor (PNGImageDecoder for instance) and not calling end() of the inflater. Well, if you pass in the c-tor w/ just new, no chance... and yes calling close() on the stream does not close the inflater if it's manually passed as c-tor parameter. This is not a true leak since it'd be released by the finalizer... when it deems it necessary. Till that moment it eats native memory so badly it can cause linux oom_killer to kill the process with impunity. The main issue is that finalization in java is very unreliable and G1 made it worse till 7.0.2. Moral of the story: release native resources as soon as you can, the finalizer is just too poor.
Same case w/ java.util.zip.Deflater, this one is far worse since Deflater is memory hungry in java, i.e. always uses 15bits (max) and 8memory level (9 is max) allocating several hundreds KB of native mem. Fortunately, Deflater is not widely used and to my knowledge JDK contains no misuses. Always call end() if you manually create a Deflater or Inflater. The best part of the last two: you can't find them via normal profiling tools available.
(I can add some more time wasters I have encountered upon request)
Good luck and stay safe, leaks are evil!
The following is a pretty pointless example, if you do not understand JDBC. Or atleast how JDBC expects a developer to close Connection, Statement and ResultSet instances before discarding them or losing references to them, instead of relying on the implementation of finalize.
ReplyDeletevoid doWork()
{
try
{
Connection conn = ConnectionFactory.getConnection();
PreparedStatement stmt = conn.preparedStatement("some query"); // executes a valid query
ResultSet rs = stmt.executeQuery();
while(rs.hasNext())
{
... process the result set
}
}
catch(SQLException sqlEx)
{
log(sqlEx);
}
}
The problem with the above is that the Connection object is not closed, and hence the physical connection will remain open, until the garbage collector comes around and sees that it is unreachable. GC will invoke the finalize method, but there are JDBC drivers that do not implement the finalize, at least not in the same way that Connection.close is implemented. The resulting behavior is that while memory will be reclaimed due to unreachable objects being collected, resources (including memory) associated with the Connection object might simply not be reclaimed.
In such an event where the Connection's finalize method does not clean up everything, one might actually find that the physical connection to the database server will last several garbage collection cycles, until the database server eventually figures out that the connection is not alive (if it does), and should be closed.
Even if the JDBC driver were to implement finalize, it is possible for exceptions to be thrown during finalization. The resulting behavior is that any memory associated with the now "dormant" object will not be reclaimed, as finalize is guaranteed to be invoked only once.
The above scenario of encountering exceptions during object finalization is related to another other scenario that could possibly lead to a memory leak - Object resurrection. Object resurrection is often done intentionally by creating a strong reference to the object from being finalized, from another object. When object resurrection is misused it will lead to a memory leak in combination with other sources of memory leaks.
There are plenty more examples that you can conjure up - like
managing a List instance where you are only adding to the list and not deleting from it (although you should be getting rid of elements you no longer need), or
opening Sockets or Files but not closing them when they are no longer needed (similar to the above example involving the Connection class).
not unloading Singletons when bringing down a Java EE application. Apparently, the Classloader that loaded the Singleton class will retain a reference to the class, and hence the singleton instance will never be collected. When a new instance of the application is deployed, a new class loader is usually created, and the former class loader will continue to exist due to the singleton.
The answer depends entirely on what the interviewer thought they were asking.
ReplyDeleteIs it possible in practice to make Java leak? Of course it is, and there are plenty of examples in the other answers.
But there are multiple meta-questions that may have been being asked?
Is a theoretically "perfect" Java implementation vulnerable to leaks?
Does the candidate understand the difference between theory and reality?
Does the candidate understand how garbage colleciton works?
Or how garbage collection is supposed to work in an ideal case?
Do they know they can call other languages through native interfaces?
Do they know to leak memory in those other languages?
Does the candidate even know what memory management is, and what is going on behind the scese in Java?
I'm reading your meta-question as "What's an answer I could have used in this interview situation". And hence, I'm going to focus on interview skills instead of Java. I believe your more likely to repeat the situaiton of not knowing the answer to a question in an interview than you are to be in a place of needing to know how to make Java leak. So, hopefully, this will help.
One of the most important skills you can develop for interviewing is learning to activley listen to the questions and working with the interviewer to extract their intent. Not only does this let you answer their question the way they want, but also shows that you have some vital communication skills. And when it comes down to a choice between many equally talented developers, I'll hire the one who listens, thinks, and understands before they respond every time.
Any time you keep references around to objects that you no longer need you have a memory leak. See Handling memory leaks in Java programs for examples of how memory leaks manifest themselves in Java and what you can do about it.
ReplyDeleteMost examples here are "too complex". They are edge case. With theses examples, the programmer made a mystake (like don't redifining equals/hashcode), or has been biten by a corner case of the JVM/JAVA (load of class with static...). I think that's not the type of example an interviewer want or even the most common case.
ReplyDeleteBut there are really simpler case to memory leaks. Garbage collector only free what is no longer referenced. We as Java developper don't care about memory. We allocate it when needed and let it being freed automatically. Fine.
But any long lived application tend to have shared state. It can be anything, statics, singletons... Often non trivial application tend to make complex objects graphs. Just forgetting to set a reference to null or more often forgetting to remove one object from a collection is enough to make a memory leak.
Of course all sort of listeners (like UI listeners), caches, or any long lived shared state tend to produce memory leak if not properluy handled. What shall be understood is that this is not a Java corner case, or a problem with the garbage collector. It a design problem. We design that we add listener to a long lived object, but we don't remove the listener when no longer needed. We cache objects, but we have no strategy to remove them from the cache.
We maybe have a complex graph that store previous state that is needed by computation. But the previous state is itself linked to the state before and so on.
Like we have to close SQL connections or files. We need to set proper referenres to null and remove elements from collection. We shall have proper caching strategies (maximum memory size, number of elements, or timers). All object that allow listener to be notified must provide both a addListener and removeListener method. And when theses notifier are no longer used, they must clear their listener list.
MemoryLeak is indeed trully possible and is perfectly predictible. No need for special language features or corner case. Memory leaks are either an indicator that something is maybe missing or even of design problems.
Probably one of the simplest examples is the implementation of ArrayList.remove(int):
ReplyDeletepublic E remove(int index) {
RangeCheck(index);
modCount++;
E oldValue = (E) elementData[index];
int numMoved = size - index - 1;
if (numMoved > 0)
System.arraycopy(elementData, index + 1, elementData, index,
numMoved);
elementData[--size] = null; // Let gc do its work
return oldValue;
}
Would you have thought to clear the array element that is no longer used? That reference might keep a huge object alive ...
Create a static Map and keep adding hard references to it. Those will never be GC'd.
ReplyDeletepublic class Leaker {
private static final Map<String, Object> CACHE = new HashMap<String, Object>();
// Keep adding until failure.
public static void addToCache(String key, Object value) { Leaker.CACHE.put(key, value); }
}
I can copy my answer from here:
ReplyDeleteEasiest way to cause memory leak in Java?
"A memory leak, in computer science (or leakage, in this context), occurs when a computer program consumes memory but is unable to release it back to the operating system." (Wikipedia)
The easy answer is: You can't. Java does automatic memory management and will free resources that are not needed for you. You can't stop this from happening. It will ALWAYS be able to release the resources. In programs with manual memory management, this is different. You cann get some memory in C using malloc(). To free the memory, you need the pointer that malloc returned and call free() on it. But if you don't have the pointer anymore (overwritten, or lifetime exceeded), then you are unfortunately incapable of freeing this memory and thus you have a memory leak.
All the other answers so far are in my definition not really memory leaks. They all aim at filling the memory with pointless stuff real fast. But at any time you could still dereference the objects you created and thus freeing the memory --> NO LEAK. acconrad's answer comes pretty close though as I have to admit since his solution is effectively to just "crash" the garbage collector by forcing it in an endless loop).
The long answer is: You can get a memory leak by writing a library for Java using the JNI, which can have manual memory management and thus have memory leaks. If you call this library, your java process will leak memory. Or, you can have bugs in the JVM, so that the JVM looses memory. There are probably bugs in the JVM, there may even be some known ones since garbage collection is not that trivial, but then it's still a bug. By design this is not possible. You may be asking for some java code that is effected by such a bug. Sorry I don't know one and it might well not be a bug anymore in the next Java version anyway.
Take any web application running in any servlet container (Tomcat, Jetty, Glassfish, whatever...). Redeploy the app 10 or 20 times in a row (it may be enough to simply touch the WAR in the server's autodeploy directory.
ReplyDeleteUnless anybody has actually tested this, chances are high that you'll get an OutOfMemoryError after a couple of redeployments, because the application did not take care to clean up after itself. You may even find a bug in your server with this test.
The problem is, the lifetime of the container is longer than the lifetime of your application. You have to make sure that all references the container might have to objects or classes of your application can be garbage collected.
If there is just one reference surviving the undeployment of your web app, the corresponding classloader and by consequence all classes of your web app cannot be garbage collected.
Threads started by your application, ThreadLocal variables, logging appenders are some of the usual suspects to cause classloader leaks.
You able to make memory leak with sun.misc.Unsafe class. In fact this service class is used in different standart classes (for example in java.nio classes). You can't create instance of this class directly, but you may use reflection to do that.
ReplyDeleteCode doesn't compile in Eclipse IDE - compile it using command javac (during compilation you'll get warnings)
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import sun.misc.Unsafe;
public class TestUnsafe {
public static void main(String[] args) throws Exception{
Class unsafeClass = Class.forName("sun.misc.Unsafe");
Field f = unsafeClass.getDeclaredField("theUnsafe");
f.setAccessible(true);
Unsafe unsafe = (Unsafe) f.get(null);
System.out.print("4..3..2..1...");
try
{
for(;;)
unsafe.allocateMemory(1024*1024);
} catch(Error e) {
System.out.println("Boom :)");
e.printStackTrace();
}
}
}
Here's a simple/sinister one via http://wiki.eclipse.org/Performance_Bloopers#String.substring.28.29.
ReplyDeletepublic class StringLeaker
{
private final String muchSmallerString;
public StringLeaker()
{
// Imagine the whole Declaration of Independence here
String veryLongString = "We hold these truths to be self-evident...";
// The substring here maintains a reference to the internal char[]
// representation of the original string.
this.muchSmallerString = veryLongString.substring(0, 1);
}
}
Because the substring refers to the internal representation of the original, much longer string, the original stays in memory. Thus, as long as you have a StringLeaker in play, you have the whole original string in memory, too, even though you might think you're just holding on to a single-character string.
The way to avoid storing an unwanted reference to the original string is to do something like this:
...
this.muchSmallerString = new String(veryLongString.substring(0, 1));
...
For added badness, you might also .intern() the substring:
...
this.muchSmallerString = veryLongString.substring(0, 1).intern();
...
Doing so will keep both the original long string and the derived substring in memory even after the StringLeaker instance has been discarded.
Maybe by using external native code through JNI?
ReplyDeleteWith pure Java, it is almost impossible.
But that is about a "standard" type of memory leak, when you cannot access the memory anymore, but it is still owned by the application. You can instead keep references to unused objects, or open streams without closing them afterwards.
A common example of this in GUI code is when creating a widget/component and adding a listener to some static/application scoped object and then not removing the listener when the widget is destroyed. Not only do you get a memory leak but a performance hit as when whatever you are listening too fires events all your old listeners are called too.
ReplyDeleteI have had a nice "memory leak" in relation to PermGen and xml-parsing once.
ReplyDeleteThe XMl-Parser we used (cant remember which one it was) did a String.intern() on tag names, to make comparison faster.
One of our customers had the gread idea to store data values not in xml-attributes or text, but as tagnames, so we had a document like:
<data>
<1>bla</1>
<2>foo</>
...
</data>
In fact, they did not use numbers but longer textual id's (around 20chars), which where unique and came in at a rate of 10-15 Million a day. That makes 200MB of rubbish a day, which is never needed again, and never GCed (since it is in permgen). We had permgen set to 512MB, so it took around 2 days for the OOME to arrive...
I thought it was interesting that no one used the internal class examples. If you have an internal class; it inherently maintains a reference to the containing class. Of course it is not technically a memory leak because Java WILL eventually clean it up; but this can cause classes to hang around longer than anticipated.
ReplyDeletepublic class Example1 {
public Example2 getNewExample2() {
return this.new Example2();
}
public class Example2 {
public Example2() {}
}
}
Now if you call Example1 and get an Example2 discarding Example1, you will inherently still have a link to an Example1 object.
public class Referencer {
public static Example2 GetAnExample2() {
Example1 ex = new Example1();
return ex.getNewExample2();
}
public static void main(String[] args) {
Example2 ex = Referencer.GetAnExample2();
// As long as ex is reachable; Example1 will always remain in memory.
}
}
I've also heard a rumor that if you have a variable that exists for longer than a specific amount of time; Java assumes that it will always exist and will actually never try to clean it up if cannot be reached in code anymore. But that is completely unverified.
I don't think anyone has said this yet: you can resurrect an object by overriding the finalize() method such that finalize() stores a reference of this somewhere. The garbage collector will only be called once on the object so after that the object will never destroyed.
ReplyDeleteThe interviewer might have be looking for a circular reference solution:
ReplyDeletepublic static void main(String[] args) {
while (true) {
Element first = new Element();
first.next = new Element();
first.next.next = first;
}
}
This is a classic problem with reference counting garbage collectors. You would then politely explain that JVMs use a much more sophisticated algorithm that doesn't have this limitation.
-Wes Tarle
As a lot of people have suggested, Resource Leaks are fairly easy to cause - like the JDBC examples. Actual Memory leaks are a bit harder - especially if you aren't relying on broken bits of the JVM to do it for you...
ReplyDeleteThe ideas of creating objects that have a very large footprint and then not being able to access them aren't real memory leaks either. If nothing can access it then it will be garbage collected, and if something can access it then it's not a leak...
One way that used to work though - and I don't know if it still does - is to have a three-deep circular chain. As in Object A has a reference to Object B, Object B has a reference to Object C and Object C has a reference to Object A. The GC was clever enough to know that a two deep chain - as in A <--> B - can safely be collected if A and B aren't accessible by anything else, but couldn't handle the three-way chain...
I think that a valid example could be using ThreadLocal variables in an environment where threads are pooled.
ReplyDeleteFor instance, using ThreadLocal variables in Servlets to communicate with other web components, having the threads being created by the container and maintaining the idle ones in a pool. ThreadLocal variables, if not correctly cleaned up, will live there until, possibly, the same web component overwrites their values.
Of course, once identified, the problem can be solved easily.
I recently encountered a memory leak situation caused in a way by log4j.
ReplyDeleteLog4j has this mechanism called Nested Diagnostic Context(NDC) which is is an instrument to distinguish interleaved log output from different sources. The granularity at which NDC works is threads, so it distinguishes log outputs by different threads separately.
In order to store thread specific tags, log4j's NDC class uses a Hashtable which is keyed by the Thread object itself (as opposed to say the thread id), and thus till the NDC tag stays in memory all the objects that hang off of the thread object also stay in memory. In our web application we use NDC to tag logoutputs with a request id to distinguish logs from a single request separately. The container that associates the NDC tag with a thread, also removes it while returning the response from a request. The problem occured when during the course of processing a request, a child thread was spawned, something like the following code:
pubclic class RequestProcessor {
private static final Logger logger = Logger.getLogger(RequestProcessor.class);
public void doSomething() {
....
final List<String> hugeList = new ArrayList<String>(10000);
new Thread() {
public void run() {
logger.info("Child thread spawned")
for(String s:hugeList) {
....
}
}
}.start();
}
}
So an NDC context was associated with inline thread that was spawned. The thread object that was the key for this NDC context, is the inline thread which has the hugeList object hanging off of it. Hence even after the thread finished doing what it was doing, the reference to the hugeList was kept alive by the NDC context Hastable, thus causing a memory leak.
there are many different situations memory will leak. One i encountered, which expose a map that should not be exposed and used in other place.
ReplyDeletepublic class ServiceFactory {
private Map<String, Service> services;
private static ServiceFactory singleton;
private ServiceFactory() {
services = new HashMap<String, Service>();
}
public static synchronized ServiceFactory getDefault() {
if (singleton == null) {
singleton = new ServiceFactory();
}
return singleton;
}
public void addService(String name, Service serv) {
services.put(name, serv);
}
public void removeService(String name) {
services.remove(name);
}
public Service getService(String name, Service serv) {
return services.get(name);
}
// the problematic api, which expose the map.
//and user can do quite a lot of thing from this api.
//for example, create service reference and forget to dispose or set it null
//in all this is a dangerous api, and should not expose
public Map<String, Service> getAllServices() {
return services;
}
}
// resource class is a heavy class
class Service {
}
An example I recently fixed is creating new GC and Image objects, but forgetting to call dispose() method.
ReplyDeleteGC javadoc snippet:
Application code must explicitly invoke the GC.dispose() method to
release the operating system resources managed by each instance when
those instances are no longer required. This is particularly important
on Windows95 and Windows98 where the operating system has a limited
number of device contexts available.
Image javadoc snippet:
Application code must explicitly invoke the Image.dispose() method to
release the operating system resources managed by each instance when
those instances are no longer required.
One possibility is to create a wrapper for an ArrayList that only provides one method: one that adds things to the ArrayList. Make the ArrayList itself private. Now, construct one of these wrapper objects in global scope (as a static object in a class) and qualify it with the final keyword (e.g. public static final ArrayListWrapper wrapperClass = new ArrayListWrapper()). So now the reference cannot be altered. That is, wrapperClass = null won't work and can't be used to free the memory. But there's also no way to do anything with wrapperClass other than add objects to it. Therefore, any objects you do add to wrapperClass are impossible to recycle.
ReplyDeleteEveryone always forgets the native code route. Here's a simple formula for a leak:
ReplyDeleteDeclare native method.
In native method, call malloc. Don't call free.
Call the native method.
Remember, memory allocations in native code come from the JVM heap.
A few suggestions:
ReplyDeleteuse commons-logging in a servlet container (a bit provocative perhaps)
start a thread in a servlet container and don't return from it's run method
load animated gifs in a servlet container (this will start an animation thread)
The above effects could be 'improved' by redeploying the application ;)
Recently stumbled upon this:
Calling "new java.util.zip.Inflater();" without calling "Inflater.end()" ever
Read http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5072161 and linked issues for an in-depth-discussion.
You can create a moving memory leak by creating a new instance of a class in that class's finalize method. Bonus points if the finalizer creates multiple instances. Here's a simple program that leaks the entire heap in sometime between a few seconds and a few minutes depending on your heap size:
ReplyDeleteclass Leakee {
public void check() {
if (depth > 2) {
Leaker.done();
}
}
private int depth;
public Leakee(int d) {
depth = d;
}
protected void finalize() {
new Leakee(depth + 1).check();
new Leakee(depth + 1).check();
}
}
public class Leaker {
private static boolean makeMore = true;
public static void done() {
makeMore = false;
}
public static void main(String[] args) throws InterruptedException {
// make a bunch of them until the garbage collector gets active
while (makeMore) {
new Leakee(0).check();
}
// sit back and watch the finalizers chew through memory
while (true) {
Thread.sleep(1000);
System.out.println("memory=" +
Runtime.getRuntime().freeMemory() + " / " +
Runtime.getRuntime().totalMemory());
}
}
}