java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
The application dies with “java.lang.OutOfMemoryError: unable to create new native thread.” The solution is to reduce the stack size. The JVM has an interesting implementation, the design of which I don’t completely understand, but the implication is that the more memory is allocated for the heap (not necessarily used by the heap), the less memory available in the stack, and since threads are made from the stack, in practice this means more “memory” in the heap sense (which is usually what people talk about) results in less threads being able to run concurrently.
Each thread by default gets a stack of 1MB. In a highly multithreaded system (~1500 threads), it is possible that we run out of contiguous blocks of memory on that machine. The limitation is normally stack space (which must be in contiguous blocks) and since every thread consumes this scattered about you rapidly run out of contiguous blocks. Reference:http://stackoverflow.com/questions/481900/whats-the-maximum-number-of-threads-in-windows-server-2003
To reduce the stack size, add “-Xss64kb” to the JVM options. Start with 64k, try the application, then if it doesn’t work (it will fail with a Java.lang.StackOverFlowError), increase the stack to 128k, then 256k, and so on. The default stack size is 8192k so there’s a wide range to test.
Also, on Linux, you’ll need to set the Linux thread stack size to the same value as the JVM stack size to get full benefits. To do that, use “ulimit -s <size in kb>”. Note that the stack size applies per user, so you have to modify the init script, or edit /etc/security/limits.conf (on Debian/Ubuntu at least).
GemFire is a highly multi-threaded system and at any given point in time there are multiple thread pools and threads that are in use. The default stack size setting for a thread in java is 1MB. Stack size has to be allocated in contiguous blocks and if the machine is being used actively and there are many threads running in the system (Task Manager shows the number of active threads), you may encounter OutOfMemoryError: unable to create new native thread , even though your process has enough available heap. If this happens, consider reducing the stack size requirement for threads on the cache server. The stack size setting for cache servers can be set using the -Xss vm arg. This can be set to 384k or 512k in such cases
