topic How to parse and filter proofpoint list in General Topics

How to parse and filter proofpoint list

MMeld_Testing — Mon, 22 Feb 2021 16:13:30 GMT

Hello,

Since the default proofpoint miner is not working 'im trying to find a workaround to be able to download and filter the lists.

I have tried to use the generic json or csv miner but i'm having issues with both:

With the json miner I receive 0 values from the lists, I guess is not able to parse it.

This is an example from the proofpoint list:

{

"webmail.bokep-indo.grup-whatsapp.xyz" : {

"Drop" : "72"

"beaconsupport.com" : {

"RemoteAccessService" : "51"

"fbgaragedoors.com" : {

"Drop" : "37"

"webmail.marelanhostlivev2.event-op.cf" : {

"AbusedTLD" : "98"

}

With the minemeld.ft.csv.CSVFT miner I have partial success, all the fields are parsed but filtering on "score" is not working.

Here is a log example

{
"category": "1",
"_age_out": 1616595084032,
"confidence": 80,
"share_level": "red",
"_last_run": 1614010284043,
"sources": [
"ProofpointET"
],
"score": "117",
"first_seen": 1614003084032,
"type": "IPv4",
"last_seen": 1614003084032
}

If in output condition I add "score > 99" this is ignored or not matched

Do you have any suggestion on how to manage a custom list?

The originali list is a simple csv/json/txt with indicator,category,score.

Thank you

Re: How to parse and filter proofpoint list

lmori — Tue, 23 Feb 2021 08:40:37 GMT

For that JSON structure you cannot use the JSON parser, as the indicators are the keys of the feed and this is not supported. The JSON Miners expects a list of objects/indicators. Also if the feed is large JSON does not scale too well, as the full file should be loaded and parsed to extract indicators. I would suggest to use CSV Miner instead.

The problem with the filter on the score is that the score attribute is a string and not a number, the filter score > 99 could not work then. You should convert score to number before filtering, basically yous should try: to_number(score) > 99

Re: How to parse and filter proofpoint list

MMeld_Testing — Thu, 25 Feb 2021 08:52:52 GMT

It worked, thank you for your reply.

Is there a list of functions that can be used as filter in the output node?

Thank you.