IP addresses disappearing from miner

Reply
L2 Linker

IP addresses disappearing from miner

Hello,

 

I've created a miner to add IP address based on stdlib.listIPv4Generic and class minemeld.ft.local.YamlIPv4FT. Default configuration (just cloned).

 

This week, some IP addresses just disappeared from this miner.

We added an IP address to the miner last 21st, on the 22nd of this month the IP address disappeared from the miner. Older indicators were not removed from the miner.

When I checked the logs, I found a EMIT_WITHDRAW event:

{

    "comment": " Win32.Conficker.Cp2p",

    "_age_out": 1511357349093,

    "confidence": 100,

    "_withdrawn": 1511357349181,

    "direction": "inbound",

    "share_level": "red",

    "_last_run": 1511278036128,

    "sources": [

        "feed_IPv4"

    ],

    "first_seen": 1511277006844,

    "type": "IPv4",

    "last_seen": 1511277006844

}

 

Here is the EMIT_UPDATE event for this indicator:

{

    "comment": "Win32.Conficker.Cp2p",

    "_age_out": 4294967295000,

    "confidence": 100,

    "direction": "inbound",

    "share_level": "red",

    "_last_run": 1511277006844,

    "sources": [

        "feed_IPv4"

    ],

    "first_seen": 1511277006844,

    "type": "IPv4",

    "last_seen": 1511277006844

}

 

 

 

 

 

Here is another  IP address that had the same issue:

{

    "comment": "North Korean Trojan: Volgmer",

    "_age_out": 1511181387647,

    "confidence": 100,

    "_withdrawn": 1511181387873,

    "share_level": "red",

    "_last_run": 1511027230747,

    "sources": [

        "feed_IPv4"

    ],

    "first_seen": 1510845540625,

    "type": "IPv4",

    "last_seen": 1510845540625

}

 

Apparently, 200 IPv4 were removed from this miner and I can’t understand why. As I mentioned, older indicators were not removed from the miner.

 

Can you please help?

I am attaching here all events involved in one indicator that went away from the miner.

L2 Linker

Re: IP addresses disappearing from miner

I understand now what happened. It is a bug in MineMeld (in my opinion)

 

Let’s say you have two users connected to MineMeld.

 

The first user opens one miner to add an indicator to it. At the same time, a second user opens the same miner. So, they are listing the same indicators.

Now, let’s say that the first user adds an indicator to the miner, but the second user doest refresh the page and adds an indicator just after the first user.

 

It happens that MineMeld will withdraw the first indicator added by the first user because the browser of the last user (second user) didn’t list that first indicator.

I did the test here and I could reproduce this issue.

 

 

So, I guess someone in my company left MineMeld opened and didn’t refresh the browser before adding indicators to MineMeld.  We lost 200 indicators because of that.

 

 

 

 Can someone from MineMeld team confirm that this is a bug?

L7 Applicator

Re: IP addresses disappearing from miner

Hi @alyssonalmeida,

yes, it's a limitation/bug for the local Miner - something we are phasing out. There is already a new version of the local Miner called "stdlib.localDB" Miner where this limitation has been removed, with additional features (TTL per indicator) and improved API and scalability. My suggestion is to switch from the old local Miner to the new one.

 

luigi

L2 Linker

Re: IP addresses disappearing from miner

Thank you, @lmori.

 

Is there any easy way to export the indicators from the old miner to the new one?

Trying to save time here :)

 

Is it possible to import a CSV into the miner?

 

L2 Linker

Re: IP addresses disappearing from miner

L7 Applicator

Re: IP addresses disappearing from miner

Yes, that's exactly the best way.

L0 Member

Re: IP addresses disappearing from miner

With the stdlib.localDB miner what is the default age out setting? I do not see it in the config. Is it possible to adjust the default age out for indicators that do not come in with a ttl?

L5 Sessionator

Re: IP addresses disappearing from miner

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!