"Top Domains" report?

[networkadmin]

I think I've asked this but can't find the thread.  What is the recommended way to get some kind of "Top Domains Visited" report?

Everything seems to default to rdns of the endpoint which is not much use sadly with cloud/online content providers i.e. your users visit Amazon/BBC but the resolved destination is node-17-cluster-5.eu.akamai.com and so on.

Thanks.

[swhyte]

I believe this is the thread you were looking for (let me know if this helps):

Reports - Best way to see top URLs...

networkadmin

28 posts since

Feb 12, 2010

I'm struggling a little with the documentation on how to generate useful reports.

If I look in the ACC or default reports I can see destinations but they are simply

a mix of raw hostname and rdns lookups - they might show a lot of traffic to, say,

a88-221-183-148.deploy.akamaitechnologies.com, but they won't show that traffic was

actually people looking at

http://news.bbc.co.uk

.

How can I get a report that (for example) simply show the top X sites (

not

individual pages)

visited for the past X hour or days please?

Also I'm unclear what I need to enable in terms of logging to be able to do this - do I need to

enable (as a minimum) alerting on all URLs for a URL profile assigned to a policy, or does

the PAN log all this info somewhere by default?

Thanks!

helenio.sartori

7 posts since

Jan 4, 2010

Reply 1. Re: Reports - Best way to see top URLs visited?

Feb 22, 2010 1:29 AM

I'm also looking for the same issue .. I'd like to produce report based on URL domain and

not only hits but also volume of traffic for this domains. Till now I wasn't able to do it ... is

that on the road map ?

nrice

60 posts since

Nov 30, 2009

Reply 2. Re: Reports - Best way to see top URLs visited?

Feb 22, 2010 1:56 PM

Reports - Best way to see top URLs...

Generated by Jive SBS on 2010-03-23-05:00

2

The Reports don't include an option to view the top X domains visited. To view the top

X URLs you can create a custom report in which you'd choose the "URL Log" as the

Database, choose "URL" as one of your options under "Columns", choose the top X option

you'd like and the period of data. To see traffic in the log, either the URL itself or the URL

category must be set to alert. Traffic that is allowed and not flagged in any way, will not be

recorded in the logs.

Nancy Rice

Technical Support

Palo Alto Networks

1-866-898-9087

networkadmin

28 posts since

Feb 12, 2010

Reply 3. Re: Reports - Best way to see top URLs visited?

Feb 23, 2010 7:48 AM

in response to:

nrice

Thanks Nancy.

Are there any plans to change this please?

I ask as, respectfully, there are lots of reports by default which don't seem overly relevant

(admittedly I only speak for myself here) yet this seems to me to be a fairly fundamental

"What's our Internet connection being used for?" report, IYSWIM?

Thanks.

nrice

60 posts since

Nov 30, 2009

Reply 4. Re: Reports - Best way to see top URLs visited?

Feb 23, 2010 12:39 PM

in response to:

networkadmin

I'll submit a request for the reporting features mentioned in this string.

Reports - Best way to see top URLs...

Generated by Jive SBS on 2010-03-23-05:00

3

Nancy Rice

Technical Support

Palo Alto Networks

1-866-898-9087

[networkadmin]

That'd be the one!

So any update on the feature request please?

[networkadmin]

Bump.. anyone please?

I'm assuming this isn't in 3.1?

[RISI]

Anything?

[Remo]

@RISI

Unfortunetely not. For a top domain report you still need something else. With splunk for example such a report is possible.

[BPry]

@Remo @RISI @networkadmin,

Forgive me if this was already offered, as I didn't read the longer post about what was already suggested. However, this is possible if you have a lot of log space to work with directly from the firewall and the patience to script a bit of the results. 

So you could actually set the URL-Filtering policy so that all categories have an 'alert' or better status, therefore nothing gets set to 'allow' as you wouldn't get the logs. You will then get logs anytime someone visits bbc.co.uk or amazon.com or whathave you for example. 

The reporting is where it would get trickier as loading 1 'www.amazon.com' for example generates a large amount of further URLs to be logged when it fetches content. Generally however if you sourt by count specifically the actual URL log for 'www.amazon.com' is going to have more hits then 'pushy-service-us-west-2.prod.aws.lcloud' for example. This isn't a perfect solution, hell it's not even that good of one, but it would technically work. 

The primary issue that you'll run into is that this will generate a lot of logs, and from a peer storage allocation perspective I can't recommend that you really do it. It would still be drastically easier to simply offload this to something like splunk that can strip out all the fluff that you don't care about for you. 

[Remo]

@BPry

Yes, the scripting solution works. The criticism that PaloAlto must accept is really about this "Domain" report. At least in my environment (we log EVERYTHING) I have almost 0 urls like www.amazon.com, www.google.com, www.paloaltonetworks.com and so on. In every log there is something after the domain like www.amazon.com/anything/anything/somefile.html. so when you want to create a script you need to split the URL at the first "/" to get the domain and then count these entries to get the hits to a particular domain. Other solutions/vendors do this out of the box (which actually shouldn't be really hard for paloalto to implement)... In addition with other solutions it is also possible to get a report with the amount of traffic to specific domains, which is also not possible with paloalto without a not so simple script or without something like splunk.

Sometimes I don't really know what to think: one one hand there are some "basics" missing, which would be great for a lot of customers and on the other hand I love the API - if something isn't built in at least PaloAlto gives us the possibility do implement it by ourselfs. Yes, this means some work, but you then also get exactly what you need instead of buying another (expensive) product which then fits 80% of your needs (instead of lets say 60% without some scripting) ... advantages and disadvantages - we will always have to live with them