topic Re: IP addresses disappearing from miner in General Topics

IP addresses disappearing from miner

alyssonalmeida — Sat, 25 Nov 2017 14:34:06 GMT

Hello,

I've created a miner to add IP address based on stdlib.listIPv4Generic and class minemeld.ft.local.YamlIPv4FT. Default configuration (just cloned).

This week, some IP addresses just disappeared from this miner.

We added an IP address to the miner last 21st, on the 22^nd of this month the IP address disappeared from the miner. Older indicators were not removed from the miner.

When I checked the logs, I found a EMIT_WITHDRAW event:

{

"comment": " Win32.Conficker.Cp2p",

"_age_out": 1511357349093,

"confidence": 100,

"_withdrawn": 1511357349181,

"direction": "inbound",

"share_level": "red",

"_last_run": 1511278036128,

"sources": [

"feed_IPv4"

"first_seen": 1511277006844,

"type": "IPv4",

"last_seen": 1511277006844

}

Here is the EMIT_UPDATE event for this indicator:

{

"comment": "Win32.Conficker.Cp2p",

"_age_out": 4294967295000,

"confidence": 100,

"direction": "inbound",

"share_level": "red",

"_last_run": 1511277006844,

"sources": [

"feed_IPv4"

"first_seen": 1511277006844,

"type": "IPv4",

"last_seen": 1511277006844

}

Here is another IP address that had the same issue:

{

"comment": "North Korean Trojan: Volgmer",

"_age_out": 1511181387647,

"confidence": 100,

"_withdrawn": 1511181387873,

"share_level": "red",

"_last_run": 1511027230747,

"sources": [

"feed_IPv4"

"first_seen": 1510845540625,

"type": "IPv4",

"last_seen": 1510845540625

}

Apparently, 200 IPv4 were removed from this miner and I can’t understand why. As I mentioned, older indicators were not removed from the miner.

Can you please help?

I am attaching here all events involved in one indicator that went away from the miner.

Re: IP addresses disappearing from miner

alyssonalmeida — Sat, 25 Nov 2017 15:49:03 GMT

I understand now what happened. It is a bug in MineMeld (in my opinion)

Let’s say you have two users connected to MineMeld.

The first user opens one miner to add an indicator to it. At the same time, a second user opens the same miner. So, they are listing the same indicators.

Now, let’s say that the first user adds an indicator to the miner, but the second user doest refresh the page and adds an indicator just after the first user.

It happens that MineMeld will withdraw the first indicator added by the first user because the browser of the last user (second user) didn’t list that first indicator.

I did the test here and I could reproduce this issue.

So, I guess someone in my company left MineMeld opened and didn’t refresh the browser before adding indicators to MineMeld. We lost 200 indicators because of that.

Can someone from MineMeld team confirm that this is a bug?

Re: IP addresses disappearing from miner

lmori — Sat, 25 Nov 2017 16:20:21 GMT

Hi @alyssonalmeida,

yes, it's a limitation/bug for the local Miner - something we are phasing out. There is already a new version of the local Miner called "stdlib.localDB" Miner where this limitation has been removed, with additional features (TTL per indicator) and improved API and scalability. My suggestion is to switch from the old local Miner to the new one.

luigi

Re: IP addresses disappearing from miner

alyssonalmeida — Sat, 25 Nov 2017 18:43:16 GMT

Thank you, @lmori.

Is there any easy way to export the indicators from the old miner to the new one?

Trying to save time here 🙂

Is it possible to import a CSV into the miner?

Re: IP addresses disappearing from miner

alyssonalmeida — Sat, 25 Nov 2017 18:53:23 GMT

I found this and I think that will be useful

https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/MineMeldArticles/article-id/151

Re: IP addresses disappearing from miner

lmori — Tue, 26 Dec 2017 07:37:52 GMT

Yes, that's exactly the best way.

Re: IP addresses disappearing from miner

RodneyBeaudry — Thu, 07 Jun 2018 17:34:58 GMT

With the stdlib.localDB miner what is the default age out setting? I do not see it in the config. Is it possible to adjust the default age out for indicators that do not come in with a ttl?

Re: IP addresses disappearing from miner

xhoms — Thu, 07 Jun 2018 21:37:39 GMT

Hi @RodneyBeaudry,

it defaults to 1800.

https://github.com/PaloAltoNetworks/minemeld-core/minemeld/ft/localdb.py#L47