08-13-2018 12:31 PM
Hi guys,
recently I've noticed a strange behaviour in my DShield miner (dshield.block prototype, https://www.dshield.org/block.txt). I have 2 MM (0.9.46 and 0.9.48), and both present the problem. Time to time, I can't precise the period, in fact it is not regular, my miner presents 0 indicators mined.
First, I would like to know if somebody else experienced it. Second, how you dealed with it?
Thank you in advanced.
Best regards.
09-11-2018 07:17 AM
Hi guys,
pretty sure the problem was in DShield side. I didn't change anything in my MM Machine or configuration and the behaviour returned to normal ( the same way it began its erroneous behaviour). Suddenly.
I'm closing this topic.
Best regards.
08-13-2018 12:57 PM
Hi @danilo.souza,
could you check the logs? Did the Miner polled 0 valid indicatos from the feed?
08-14-2018 06:35 AM
Hi @lmori
Look the images attached. In the first one my panel shows 0 indicators to dshield miner. The second one, shows the log. At 10hs it withdraws the indicators but don't emmit the new ones. It stays almost 30 min with 0 indicators.
Any idea.
Thank you.
08-14-2018 07:26 AM
Hi @danilo.souza,
my fault, I meant the engine logs (System > Engine > Logs). Could you check for errors on the dshield miner?
08-14-2018 07:53 AM
Hi @lmori
this is what I got, recently, related to Dshield miner.
2018-08-14T11:16:47 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 poll
2018-08-14T11:16:47 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:16:47 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
.
.
.
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 sudden_death
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 age_out
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 gc
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2018-08-14T11:17:42 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256262787 age_out
2018-08-14T11:17:42 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 poll
2018-08-14T11:27:06 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:27:06 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 sudden_death
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 age_out
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 gc
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
.
.
.
2018-08-14T11:30:33 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257033869 age_out
2018-08-14T11:30:33 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
.
.
.
2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
You can see the "Deleted in scan of _last_run: 20".
I could not put here the whole log file, too big. It helps?
Thank you.
08-14-2018 07:57 AM
Hi @lmori
I put a respost here, but it was calassified as "Spam"! How to deal with it? Was it because I wrote down some log lines?
Thank you.
08-14-2018 07:58 AM
Hi @lmori
this is what I got, recently, related to Dshield miner.
2018-08-14T11:16:47 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 poll
2018-08-14T11:16:47 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:16:47 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
.
.
.
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 sudden_death
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 age_out
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 gc
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2018-08-14T11:17:42 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256262787 age_out
2018-08-14T11:17:42 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 poll
2018-08-14T11:27:06 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:27:06 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 sudden_death
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 age_out
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 gc
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
.
.
.
2018-08-14T11:30:33 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257033869 age_out
2018-08-14T11:30:33 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
.
.
.
2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
You can see the "Deleted in scan of _last_run: 20".
I could not put here the whole log file, too big. It helps?
Thank you.
08-14-2018 07:59 AM
Hi @danilo.souza,
never happened to me before, sorry about that. Are you on the Slack community? Could you PM the logs there?
Thanks,
luigi
08-14-2018 08:02 AM
Hi
this is what I got, recently, related to Dshield miner.
2018-08-14T11:16:47 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 poll
2018-08-14T11:16:47 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:16:47 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
.
.
.
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 sudden_death
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 age_out
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 gc
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2018-08-14T11:17:42 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256262787 age_out
2018-08-14T11:17:42 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
.
.
.
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 poll
2018-08-14T11:27:06 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:27:06 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 sudden_death
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 age_out
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 gc
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
.
.
.
2018-08-14T11:30:33 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257033869 age_out
2018-08-14T11:30:33 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
.
.
.
2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
You can see the "Deleted in scan of _last_run: 20".
I could not put here the whole log file, too big. It helps?
Thank you.
08-14-2018 09:29 AM
Hi @lmori
this is what I got, recently, related to Dshield miner.
2018-08-14T11:16:47 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 poll
2018-08-14T11:16:47 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:16:47 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
...
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 sudden_death
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 age_out
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 gc
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2018-08-14T11:17:42 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256262787 age_out
2018-08-14T11:17:42 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
...
2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
...
2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
...
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 poll
2018-08-14T11:27:06 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:27:06 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 sudden_death
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 age_out
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 gc
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
...
2018-08-14T11:30:33 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257033869 age_out
2018-08-14T11:30:33 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
...
2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
You can see the "Deleted in scan of _last_run: 20".
I could not put here the whole log file, too big. It helps?
Thank you.
08-15-2018 06:08 AM
Hi @lmori
I'm attaching a txt. Let me know if you can see it.
This is what I got, recently, related to Dshield miner. You can see the "Deleted in scan of _last_run: 20". It helps?
Best regards.
08-21-2018 04:31 AM
Hi guys,
Any idea? Any help with this issue?
Thank you.
Best regards.
08-21-2018 11:11 AM
if you're using the default age out policy for the dshield prototype (default: null, interval: 257, sudden_death: true) then the only reason for the node to withdraw all indicators is a successfull poll returning zero indicators.
The dshield prototype uses the HttpFT class which parses the HTML page received from https://www.dshield.org/block.txt. A SSL decryption page between the MineMeld instance and Internet might provide a valid HTML page (captive portal) resulting in zero indicators after the parsing stage.
Is that a feasible explanation?
08-24-2018 06:38 AM
Hi @xhoms
thank you for the reply. In truth, my MM VM is in a zone that demands/show the Captive Portal when the Firewall is not capable to identify the user. However, I'm not able to asure that this is the problem. I have a very stable environment (don't remember any change in the Authentication Policies) and it was working fine until few days before I reported it here (sorry, that was when I noticed it, may be the problem was ocurring even earlier). But, if nobody else experimented it, the problem is in my environment.
Any news I will share with you. Any help I would appreciate.
Best regards.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
