How to parse and filter proofpoint list

Announcements

Changes to the LIVEcommunity experience are coming soon... Here's what you need to know.

Reply
MMeld_Testing
L0 Member

How to parse and filter proofpoint list

Hello,

Since the default proofpoint miner is not working 'im trying to find a workaround to be able to download and filter the lists.

I have tried to use the generic json or csv miner but i'm having issues with both:

 

With the json miner I receive 0 values from the lists, I guess is not able to parse it.

This is an example from the proofpoint list:

{

   "webmail.bokep-indo.grup-whatsapp.xyz" : {

      "Drop" : "72"

   },

   "beaconsupport.com" : {

      "RemoteAccessService" : "51"

   },

   "fbgaragedoors.com" : {

      "Drop" : "37"

   },

   "webmail.marelanhostlivev2.event-op.cf" : {

      "AbusedTLD" : "98"

   }

 

 

 

 

 

With the minemeld.ft.csv.CSVFT miner I have partial success, all the fields are parsed but filtering on "score" is not working.

Here is a log example

{
"category": "1",
"_age_out": 1616595084032,
"confidence": 80,
"share_level": "red",
"_last_run": 1614010284043,
"sources": [
"ProofpointET"
],
"score": "117",
"first_seen": 1614003084032,
"type": "IPv4",
"last_seen": 1614003084032
}

 

If in output condition I add "score > 99" this is ignored or not matched

 

Do you have any suggestion on how to manage a custom list?

The originali list is a simple csv/json/txt with indicator,category,score.

 

Thank you

 


Accepted Solutions
lmori
L1 Bithead

For that JSON structure you cannot use the JSON parser, as the indicators are the keys of the feed and this is not supported. The JSON Miners expects a list of objects/indicators. Also if the feed is large JSON does not scale too well, as the full file should be loaded and parsed to extract indicators. I would suggest to use CSV Miner instead.

The problem with the filter on the score is that the score attribute is a string and not a number, the filter score > 99 could not work then. You should convert score to number before filtering, basically yous should try: to_number(score) > 99

View solution in original post


All Replies
lmori
L1 Bithead

For that JSON structure you cannot use the JSON parser, as the indicators are the keys of the feed and this is not supported. The JSON Miners expects a list of objects/indicators. Also if the feed is large JSON does not scale too well, as the full file should be loaded and parsed to extract indicators. I would suggest to use CSV Miner instead.

The problem with the filter on the score is that the score attribute is a string and not a number, the filter score > 99 could not work then. You should convert score to number before filtering, basically yous should try: to_number(score) > 99

View solution in original post

MMeld_Testing
L0 Member

It worked, thank you for your reply.

 

Is there a list of functions that can be used as filter in the output node?

 

Thank you.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!