Custom Report - Maximum Limits

[Jason.Hill]

I'm currently trying to produce a custom report and i'm hitting a few walls trying to generate useful data - I'm hoping someone here might have some creative ideas, or even point out the obvious ones I may have missed.

We currently have a blacklist of source IP's which relate to brute force threats which we use in a threat rule, as we see new 'attacks' we add the sources to this list. However as a lot of these sources can indeed be legitimate hosts or gateways, we don't want to block them indefinitely.

What we want to do is run a report on a specific threat rule every 24 hours and create a list of source IP's (with duplicates removed). We'd then compare this to our blacklist and if we see a particular source ip is no longer a potential threat we would then remove that source from our black list to avoid blocking any legitimate traffic.

The current problem is any way I try to run the report and export data we easily surpass the maximum logged events in a 24 hour period - however most of these will be the same source for a 10,000+ events

Is there ANY way I can create a report for a threat rule, just showing unique source ip's and removing duplicates? If that makes sense. I just want to know there were 800 unique sources trying to brutforce and this is the list of IP's.

I've tried filtering and calving up the data by subnet range but that's getting too messy and time consuming.. everything else either hits the sort by 500 limit or the 1,000,000+ maximum row CSV limit.. ALL are still filled with hundreds of duplicate source ips so I don't get all the data regardless.

I'm at a bit of a loss so any advice or pointing in the right direction would be great.

Thanks!

[jvalentine]

There's no getting around the on-box 500 limit today.  If you want more than 500, you should contact your Palo Alto Networks SE and ask for a feature request.  For more than 500, you'll need to go off-box.  More on that later.  

You can aggregate the source-IPs in custom reporting.  I don't have much in the way of brute-forcing, so my custom report query is looking for threat type scan... but you can use (threatid eq xxxxx) if there's a particular IPS signature you're interested in, or some other variable that works for you.  

Make a new custom report, use the Threat Summary database, make sure Selected Columns have only Source address and Count.  Pick a short timeframe first (15 minutes, 1 hour, etc.) just to make sure it looks like what you're expecting.  (I expanded mine to 30-days because a single day's scan traffic is underwhelming).  Modify the query to your environment.  

As you can see, the firewall de-duplicates and then provides a count for each source address.  

If the 500-limit is interferring with your ability to address this requirement, I suggest looking into the free MineMeld (https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld) tool.  It should be able to ingest threat syslogs from your firewall, look for certain events, populate the offender into a dynamic address object group (which is already referenced in your security policy with a "deny" action), and then remove the offender after a configurable time limit.  It'll require some legwork, but probably not anywhere near as much as trying to automate the report, ingest the report, build the object group, etc. from scratch.