02-22-2021 08:13 AM
Hello,
Since the default proofpoint miner is not working 'im trying to find a workaround to be able to download and filter the lists.
I have tried to use the generic json or csv miner but i'm having issues with both:
With the json miner I receive 0 values from the lists, I guess is not able to parse it.
This is an example from the proofpoint list:
{
"webmail.bokep-indo.grup-whatsapp.xyz" : {
"Drop" : "72"
},
"beaconsupport.com" : {
"RemoteAccessService" : "51"
},
"fbgaragedoors.com" : {
"Drop" : "37"
},
"webmail.marelanhostlivev2.event-op.cf" : {
"AbusedTLD" : "98"
}
With the minemeld.ft.csv.CSVFT miner I have partial success, all the fields are parsed but filtering on "score" is not working.
Here is a log example
{
"category": "1",
"_age_out": 1616595084032,
"confidence": 80,
"share_level": "red",
"_last_run": 1614010284043,
"sources": [
"ProofpointET"
],
"score": "117",
"first_seen": 1614003084032,
"type": "IPv4",
"last_seen": 1614003084032
}
If in output condition I add "score > 99" this is ignored or not matched
Do you have any suggestion on how to manage a custom list?
The originali list is a simple csv/json/txt with indicator,category,score.
Thank you
02-23-2021 12:40 AM
For that JSON structure you cannot use the JSON parser, as the indicators are the keys of the feed and this is not supported. The JSON Miners expects a list of objects/indicators. Also if the feed is large JSON does not scale too well, as the full file should be loaded and parsed to extract indicators. I would suggest to use CSV Miner instead.
The problem with the filter on the score is that the score attribute is a string and not a number, the filter score > 99 could not work then. You should convert score to number before filtering, basically yous should try: to_number(score) > 99
02-23-2021 12:40 AM
For that JSON structure you cannot use the JSON parser, as the indicators are the keys of the feed and this is not supported. The JSON Miners expects a list of objects/indicators. Also if the feed is large JSON does not scale too well, as the full file should be loaded and parsed to extract indicators. I would suggest to use CSV Miner instead.
The problem with the filter on the score is that the score attribute is a string and not a number, the filter score > 99 could not work then. You should convert score to number before filtering, basically yous should try: to_number(score) > 99
02-25-2021 12:52 AM
It worked, thank you for your reply.
Is there a list of functions that can be used as filter in the output node?
Thank you.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
