Too many open files error in engine log

[DanWoodruff]

Hi,

As of yesterday morning I'm getting an error in my MineMeld engine log that is indicating too many open files. Also, some of my input and output miners are not updating (they show 0 entries even though there are indicators in the input table in the web interface). The timing indicates these errors are possibly the cause. Examples:

2016-10-23T07:43:43 (23610)basepoller._run ERROR: Exception in polling loop for <feed name>: IO error: <feed name>/194800.ldb: Too many open files
IOError: IO error: <feed name>/194800.ldb: Too many open files
OSError: [Errno 24] Too many open files: '/opt/minemeld/local/config/<feed name>_indicators.yml.lock'

I've restarted MineMeld and the error has gone away for now. It was only running for about 3 days before the errors started:

2016-10-20T09:28:18 (23604)launcher.main INFO: Starting mm-run.py version 0.9.24

Any guidance on how to best increase the max number of file descriptors for MineMeld?

Thanks,

Dan

[lmori]

Hi Dan,

do you see other ERRORs in the log file ?

Luigi

[DanWoodruff]

The only other things I see (grepping for error) are like this:

2016-10-23T07:39:03 (23610)basepoller._age_out_run ERROR: Exception in _age_out_loop
2016-10-23T07:39:05 (23610)local._build_iterator ERROR: <feed name> - exception loading indicators list

[DanWoodruff]

Also I should note I'm on Ubuntu 14.04.5 that I installed myself, not the OVA. Thanks!

[DanWoodruff]

I'm keeping an eye on open file descriptors by process and will report back when/if the error occurs again and what process is consuming the most files. 

$ for p in `ls /proc/*/fd -d`; do  echo "`sudo ls $p | wc -l` : $p"; done | sort -n | grep -v '^0' | tail
14 : /proc/5290/fd
15 : /proc/19530/fd
16 : /proc/1103/fd
16 : /proc/533/fd
19 : /proc/18286/fd
24 : /proc/2029/fd
64 : /proc/1845/fd
96 : /proc/5296/fd
106 : /proc/5295/fd
109 : /proc/1748/fd

[DanWoodruff · Accepted solution]

I believe this may have been an artifact of my scripted backend updates to miners and MineMeld polling each miner for changes every 1 minute. I've since created new miners to be used with these scripted updates that have a once a day update interval, rather than once a minute, relying on my script to send a HUP to the feed after it has been updated as advised in another thread by Luigi. It has been running for 24 hours with no sign of the number of open files running away (and much better CPU usage). 

[lmori]

Great !  Thanks for letting us know ! I think the original problem was a descriptor leak generated by the heavy usage of the local list Miner. It will be fixed in 0.9.28.

Luigi