Wednesday, 27 March 2013

“Too many open files” error in highly multithreaded application

Caused by: errors.txt (Too many open files)
    at Method)
    at hydra.FileUtil.appendToFile(
    ... 5 more

If we see “Too many open files” error in highly multi- threaded application, this means we have reached to maximum number of file handles.

To see where you are at with file handles on Linux:

Run the command

sysctl fs.file-nr

From this command you will get 3 numbers. First is the number of used file descriptor the second is the number of allocated but not used file descriptor and the last is the system max file descriptor.

To up this limit:
You will need to add something similar to the following in the file /etc/sysctl.conf:

fs.file-max = 204708

Also there is an individual users file handle limit, that is where the "ulimit -n" command. If you are running to this limit you will have to add entries to the /etc/security/limits.conf file for the user like the following:

@users hard nofile 81920
@users soft nofile 8192
@users hard nproc unlimited
@users soft nproc 501408
@users soft core unlimited
@users hard core unlimited

Apply the changes using :

sysctl -a