topic Re: DShield list in General Topics

DShield list

danilo.souza — Mon, 13 Aug 2018 19:31:42 GMT

Hi guys,

recently I've noticed a strange behaviour in my DShield miner (dshield.block prototype, https://www.dshield.org/block.txt). I have 2 MM (0.9.46 and 0.9.48), and both present the problem. Time to time, I can't precise the period, in fact it is not regular, my miner presents 0 indicators mined.

First, I would like to know if somebody else experienced it. Second, how you dealed with it?

Thank you in advanced.

Best regards.

Re: DShield list

lmori — Mon, 13 Aug 2018 19:57:05 GMT

Hi @danilo.souza,

could you check the logs? Did the Miner polled 0 valid indicatos from the feed?

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 13:35:11 GMT

Hi @lmori

Look the images attached. In the first one my panel shows 0 indicators to dshield miner. The second one, shows the log. At 10hs it withdraws the indicators but don't emmit the new ones. It stays almost 30 min with 0 indicators.

Any idea.

Thank you.

Re: DShield list

lmori — Tue, 14 Aug 2018 14:26:40 GMT

Hi @danilo.souza,

my fault, I meant the engine logs (System > Engine > Logs). Could you check for errors on the dshield miner?

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 14:53:05 GMT

Hi @lmori

this is what I got, recently, related to Dshield miner.

2018-08-14T11:16:47 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 poll
2018-08-14T11:16:47 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:16:47 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org

.
.
.

2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 sudden_death
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 age_out
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:16:56 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256207149 gc
2018-08-14T11:16:56 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2018-08-14T11:17:42 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256262787 age_out
2018-08-14T11:17:42 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 poll
2018-08-14T11:27:06 (31433)basepoller._polling_loop INFO: Polling dshield_blocklist
2018-08-14T11:27:06 (31433)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 sudden_death
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _last_run: 20
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 age_out
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:27:06 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256826150 gc
2018-08-14T11:27:06 (31433)table._query_by_index INFO: Deleted in scan of _withdrawn: 0

.
.
.

2018-08-14T11:30:33 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257033869 age_out
2018-08-14T11:30:33 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False
2018-08-14T11:30:58 (31432)basepoller._huppable_wait INFO: hup is clear: False

.
.
.

2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

You can see the "Deleted in scan of _last_run: 20".

I could not put here the whole log file, too big. It helps?

Thank you.

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 14:57:35 GMT

Hi @lmori

I put a respost here, but it was calassified as "Spam"! How to deal with it? Was it because I wrote down some log lines?

Thank you.

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 14:58:51 GMT

Hi @lmori

this is what I got, recently, related to Dshield miner.

.
.
.

2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

You can see the "Deleted in scan of _last_run: 20".

I could not put here the whole log file, too big. It helps?

Thank you.

Re: DShield list

lmori — Tue, 14 Aug 2018 14:59:18 GMT

Hi @danilo.souza,

never happened to me before, sorry about that. Are you on the Slack community? Could you PM the logs there?

Thanks,

luigi

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 15:00:42 GMT

Hi @lmori

I'm not. Let me try again.

Thank you.

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 15:02:14 GMT

this is what I got, recently, related to Dshield miner.

.
.
.

2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

.
.
.

2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

You can see the "Deleted in scan of _last_run: 20".

I could not put here the whole log file, too big. It helps?

Thank you.

Re: DShield list

danilo.souza — Tue, 14 Aug 2018 16:29:49 GMT

Hi @lmori

this is what I got, recently, related to Dshield miner.

...

2018-08-14T11:21:59 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256519854 age_out
2018-08-14T11:21:59 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

...

2018-08-14T11:26:16 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534256776863 age_out
2018-08-14T11:26:16 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

...

2018-08-14T11:34:50 (31433)basepoller._actor_loop INFO: dshield_blocklist - command: 1534257290921 age_out
2018-08-14T11:34:50 (31433)table._query_by_index INFO: Deleted in scan of _age_out: 0

You can see the "Deleted in scan of _last_run: 20".

I could not put here the whole log file, too big. It helps?

Thank you.

Re: DShield list

danilo.souza — Wed, 15 Aug 2018 13:08:24 GMT

Hi @lmori

I'm attaching a txt. Let me know if you can see it.

This is what I got, recently, related to Dshield miner. You can see the "Deleted in scan of _last_run: 20". It helps?

Best regards.

Re: DShield list

danilo.souza — Tue, 21 Aug 2018 11:31:04 GMT

Hi guys,

Any idea? Any help with this issue?

Thank you.

Best regards.

Re: DShield list

xhoms — Tue, 21 Aug 2018 18:11:45 GMT

@danilo.souza,

if you're using the default age out policy for the dshield prototype (default: null, interval: 257, sudden_death: true) then the only reason for the node to withdraw all indicators is a successfull poll returning zero indicators.

The dshield prototype uses the HttpFT class which parses the HTML page received from https://www.dshield.org/block.txt. A SSL decryption page between the MineMeld instance and Internet might provide a valid HTML page (captive portal) resulting in zero indicators after the parsing stage.

Is that a feasible explanation?

Re: DShield list

danilo.souza — Fri, 24 Aug 2018 13:38:01 GMT

Hi @xhoms

thank you for the reply. In truth, my MM VM is in a zone that demands/show the Captive Portal when the Firewall is not capable to identify the user. However, I'm not able to asure that this is the problem. I have a very stable environment (don't remember any change in the Authentication Policies) and it was working fine until few days before I reported it here (sorry, that was when I noticed it, may be the problem was ocurring even earlier). But, if nobody else experimented it, the problem is in my environment.

Any news I will share with you. Any help I would appreciate.

Best regards.

Re: DShield list

danilo.souza — Tue, 11 Sep 2018 14:17:35 GMT

Hi guys,

pretty sure the problem was in DShield side. I didn't change anything in my MM Machine or configuration and the behaviour returned to normal ( the same way it began its erroneous behaviour). Suddenly.

I'm closing this topic.

Best regards.