Too many open files error in engine log

Reply
L2 Linker

Too many open files error in engine log

Hi,

 

As of yesterday morning I'm getting an error in my MineMeld engine log that is indicating too many open files. Also, some of my input and output miners are not updating (they show 0 entries even though there are indicators in the input table in the web interface). The timing indicates these errors are possibly the cause. Examples:

 

 

2016-10-23T07:43:43 (23610)basepoller._run ERROR: Exception in polling loop for <feed name>: IO error: <feed name>/194800.ldb: Too many open files
IOError: IO error: <feed name>/194800.ldb: Too many open files
OSError: [Errno 24] Too many open files: '/opt/minemeld/local/config/<feed name>_indicators.yml.lock'

I've restarted MineMeld and the error has gone away for now. It was only running for about 3 days before the errors started:

 

2016-10-20T09:28:18 (23604)launcher.main INFO: Starting mm-run.py version 0.9.24

Any guidance on how to best increase the max number of file descriptors for MineMeld?

 

Thanks,

Dan

 

 

Highlighted
L7 Applicator

Re: Too many open files error in engine log

Hi Dan,

do you see other ERRORs in the log file ?

 

Luigi

L2 Linker

Re: Too many open files error in engine log

The only other things I see (grepping for error) are like this:

 

2016-10-23T07:39:03 (23610)basepoller._age_out_run ERROR: Exception in _age_out_loop
2016-10-23T07:39:05 (23610)local._build_iterator ERROR: <feed name> - exception loading indicators list
L2 Linker

Re: Too many open files error in engine log

Also I should note I'm on Ubuntu 14.04.5 that I installed myself, not the OVA. Thanks!

L2 Linker

Re: Too many open files error in engine log

I'm keeping an eye on open file descriptors by process and will report back when/if the error occurs again and what process is consuming the most files. 

$ for p in `ls /proc/*/fd -d`; do  echo "`sudo ls $p | wc -l` : $p"; done | sort -n | grep -v '^0' | tail
14 : /proc/5290/fd
15 : /proc/19530/fd
16 : /proc/1103/fd
16 : /proc/533/fd
19 : /proc/18286/fd
24 : /proc/2029/fd
64 : /proc/1845/fd
96 : /proc/5296/fd
106 : /proc/5295/fd
109 : /proc/1748/fd
L2 Linker

Re: Too many open files error in engine log

I believe this may have been an artifact of my scripted backend updates to miners and MineMeld polling each miner for changes every 1 minute. I've since created new miners to be used with these scripted updates that have a once a day update interval, rather than once a minute, relying on my script to send a HUP to the feed after it has been updated as advised in another thread by Luigi. It has been running for 24 hours with no sign of the number of open files running away (and much better CPU usage). 

L7 Applicator

Re: Too many open files error in engine log

Great !  Thanks for letting us know ! I think the original problem was a descriptor leak generated by the heavy usage of the local list Miner. It will be fixed in 0.9.28.

 

Luigi

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!