Too many open files error in engine log

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Too many open files error in engine log

L2 Linker

Hi,

 

As of yesterday morning I'm getting an error in my MineMeld engine log that is indicating too many open files. Also, some of my input and output miners are not updating (they show 0 entries even though there are indicators in the input table in the web interface). The timing indicates these errors are possibly the cause. Examples:

 

 

2016-10-23T07:43:43 (23610)basepoller._run ERROR: Exception in polling loop for <feed name>: IO error: <feed name>/194800.ldb: Too many open files
IOError: IO error: <feed name>/194800.ldb: Too many open files
OSError: [Errno 24] Too many open files: '/opt/minemeld/local/config/<feed name>_indicators.yml.lock'

I've restarted MineMeld and the error has gone away for now. It was only running for about 3 days before the errors started:

 

2016-10-20T09:28:18 (23604)launcher.main INFO: Starting mm-run.py version 0.9.24

Any guidance on how to best increase the max number of file descriptors for MineMeld?

 

Thanks,

Dan

 

 

1 accepted solution

Accepted Solutions

I believe this may have been an artifact of my scripted backend updates to miners and MineMeld polling each miner for changes every 1 minute. I've since created new miners to be used with these scripted updates that have a once a day update interval, rather than once a minute, relying on my script to send a HUP to the feed after it has been updated as advised in another thread by Luigi. It has been running for 24 hours with no sign of the number of open files running away (and much better CPU usage). 

View solution in original post

6 REPLIES 6

L7 Applicator

Hi Dan,

do you see other ERRORs in the log file ?

 

Luigi

The only other things I see (grepping for error) are like this:

 

2016-10-23T07:39:03 (23610)basepoller._age_out_run ERROR: Exception in _age_out_loop
2016-10-23T07:39:05 (23610)local._build_iterator ERROR: <feed name> - exception loading indicators list

Also I should note I'm on Ubuntu 14.04.5 that I installed myself, not the OVA. Thanks!

I'm keeping an eye on open file descriptors by process and will report back when/if the error occurs again and what process is consuming the most files. 

$ for p in `ls /proc/*/fd -d`; do  echo "`sudo ls $p | wc -l` : $p"; done | sort -n | grep -v '^0' | tail
14 : /proc/5290/fd
15 : /proc/19530/fd
16 : /proc/1103/fd
16 : /proc/533/fd
19 : /proc/18286/fd
24 : /proc/2029/fd
64 : /proc/1845/fd
96 : /proc/5296/fd
106 : /proc/5295/fd
109 : /proc/1748/fd

I believe this may have been an artifact of my scripted backend updates to miners and MineMeld polling each miner for changes every 1 minute. I've since created new miners to be used with these scripted updates that have a once a day update interval, rather than once a minute, relying on my script to send a HUP to the feed after it has been updated as advised in another thread by Luigi. It has been running for 24 hours with no sign of the number of open files running away (and much better CPU usage). 

Great !  Thanks for letting us know ! I think the original problem was a descriptor leak generated by the heavy usage of the local list Miner. It will be fixed in 0.9.28.

 

Luigi

  • 1 accepted solution
  • 5729 Views
  • 6 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!