IP addresses disappearing from miner

[alyssonalmeida]

Hello,

I've created a miner to add IP address based on stdlib.listIPv4Generic and class minemeld.ft.local.YamlIPv4FT. Default configuration (just cloned).

This week, some IP addresses just disappeared from this miner.

We added an IP address to the miner last 21st, on the 22^nd of this month the IP address disappeared from the miner. Older indicators were not removed from the miner.

When I checked the logs, I found a EMIT_WITHDRAW event:

{

    "comment": " Win32.Conficker.Cp2p",

    "_age_out": 1511357349093,

    "confidence": 100,

    "_withdrawn": 1511357349181,

    "direction": "inbound",

    "share_level": "red",

    "_last_run": 1511278036128,

    "sources": [

        "feed_IPv4"

    ],

    "first_seen": 1511277006844,

    "type": "IPv4",

    "last_seen": 1511277006844

}

Here is the EMIT_UPDATE event for this indicator:

{

    "comment": "Win32.Conficker.Cp2p",

    "_age_out": 4294967295000,

    "confidence": 100,

    "direction": "inbound",

    "share_level": "red",

    "_last_run": 1511277006844,

    "sources": [

        "feed_IPv4"

    ],

    "first_seen": 1511277006844,

    "type": "IPv4",

    "last_seen": 1511277006844

}

Here is another  IP address that had the same issue:

{

    "comment": "North Korean Trojan: Volgmer",

    "_age_out": 1511181387647,

    "confidence": 100,

    "_withdrawn": 1511181387873,

    "share_level": "red",

    "_last_run": 1511027230747,

    "sources": [

        "feed_IPv4"

    ],

    "first_seen": 1510845540625,

    "type": "IPv4",

    "last_seen": 1510845540625

}

Apparently, 200 IPv4 were removed from this miner and I can’t understand why. As I mentioned, older indicators were not removed from the miner.

Can you please help?

I am attaching here all events involved in one indicator that went away from the miner.

[alyssonalmeida]

I understand now what happened. It is a bug in MineMeld (in my opinion)

Let’s say you have two users connected to MineMeld.

The first user opens one miner to add an indicator to it. At the same time, a second user opens the same miner. So, they are listing the same indicators.

Now, let’s say that the first user adds an indicator to the miner, but the second user doest refresh the page and adds an indicator just after the first user.

It happens that MineMeld will withdraw the first indicator added by the first user because the browser of the last user (second user) didn’t list that first indicator.

I did the test here and I could reproduce this issue.

So, I guess someone in my company left MineMeld opened and didn’t refresh the browser before adding indicators to MineMeld.  We lost 200 indicators because of that.

 Can someone from MineMeld team confirm that this is a bug?

[lmori · Accepted solution]

Hi @alyssonalmeida,

yes, it's a limitation/bug for the local Miner - something we are phasing out. There is already a new version of the local Miner called "stdlib.localDB" Miner where this limitation has been removed, with additional features (TTL per indicator) and improved API and scalability. My suggestion is to switch from the old local Miner to the new one.

luigi

[alyssonalmeida]

Thank you, @lmori.

Is there any easy way to export the indicators from the old miner to the new one?

Trying to save time here 🙂

Is it possible to import a CSV into the miner?

[alyssonalmeida]

I found this and I think that will be useful

https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/MineMeldArticles/article-id/151

[lmori]

Yes, that's exactly the best way.

[RodneyBeaudry]

With the stdlib.localDB miner what is the default age out setting? I do not see it in the config. Is it possible to adjust the default age out for indicators that do not come in with a ttl?

[xhoms]

Hi @RodneyBeaudry,

it defaults to 1800.

https://github.com/PaloAltoNetworks/minemeld-core/minemeld/ft/localdb.py#L47