As of yesterday morning I'm getting an error in my MineMeld engine log that is indicating too many open files. Also, some of my input and output miners are not updating (they show 0 entries even though there are indicators in the input table in the web interface). The timing indicates these errors are possibly the cause. Examples:
2016-10-23T07:43:43 (23610)basepoller._run ERROR: Exception in polling loop for <feed name>: IO error: <feed name>/194800.ldb: Too many open files IOError: IO error: <feed name>/194800.ldb: Too many open files OSError: [Errno 24] Too many open files: '/opt/minemeld/local/config/<feed name>_indicators.yml.lock'
I've restarted MineMeld and the error has gone away for now. It was only running for about 3 days before the errors started:
2016-10-20T09:28:18 (23604)launcher.main INFO: Starting mm-run.py version 0.9.24
Any guidance on how to best increase the max number of file descriptors for MineMeld?
Solved! Go to Solution.
The only other things I see (grepping for error) are like this:
2016-10-23T07:39:03 (23610)basepoller._age_out_run ERROR: Exception in _age_out_loop 2016-10-23T07:39:05 (23610)local._build_iterator ERROR: <feed name> - exception loading indicators list
I'm keeping an eye on open file descriptors by process and will report back when/if the error occurs again and what process is consuming the most files.
$ for p in `ls /proc/*/fd -d`; do echo "`sudo ls $p | wc -l` : $p"; done | sort -n | grep -v '^0' | tail 14 : /proc/5290/fd 15 : /proc/19530/fd 16 : /proc/1103/fd 16 : /proc/533/fd 19 : /proc/18286/fd 24 : /proc/2029/fd 64 : /proc/1845/fd 96 : /proc/5296/fd 106 : /proc/5295/fd 109 : /proc/1748/fd
I believe this may have been an artifact of my scripted backend updates to miners and MineMeld polling each miner for changes every 1 minute. I've since created new miners to be used with these scripted updates that have a once a day update interval, rather than once a minute, relying on my script to send a HUP to the feed after it has been updated as advised in another thread by Luigi. It has been running for 24 hours with no sign of the number of open files running away (and much better CPU usage).
Great ! Thanks for letting us know ! I think the original problem was a descriptor leak generated by the heavy usage of the local list Miner. It will be fixed in 0.9.28.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!
The Live Community thanks you for your participation!