How URLs are logged

[helenio.sartori]

hello,

I was doing some URL reports and I did some test to see how URL are logged by the PAN FW ver 4.1.8 and I have some question.

Let's take a simple url page www.liceobellinzona.ch. when you request it, using a browser, you can see with wireshark all request used to fetch the content are:

  GET www.liceobellinzona.ch

  GET www.liceobellinzona.ch/css/default.css

  GET www.liceobellinzona.ch/css/layout.css

  GET www.liceobellinzona.ch/javascript/dtree.css

  GET www.liceobellinzona.ch/javascript/misc-functions.js

  GET www.liceobellinzona.ch/javascript/dtree.js

  GET www.liceobellinzona.ch/javascript/dtree_menu.js

  GET www.liceobellinzona.ch/images/logo1a.jpg

  GET www.liceobellinzona.ch/images/dtreemenu/join.gif

  GET www.liceobellinzona.ch/images/dtreemenu/plus.gif

  GET www.liceobellinzona.ch/images/dtreemenu/line.gif

  GET www.liceobellinzona.ch/images/dtreemenu/joinbottom.gif

  GET www.liceobellinzona.ch/libenew.jpg

Why PAN FW URL, log only the www.liceobellinzona.ch and all HTTP GET aren't ?

How URL are logged, is based on HTTP field like referrer or content type ?

[zarina]

Do you have "log container page only" option enabled in the url filtering profile? Also, you can see the list of default container pages we log: Device -> Setup -> Content-ID -> Container Pages

[dyang · Accepted solution]

The URL filtering engine will log all GET requests, assuming that you have a URL filtering profile minimally set to "alert" for that category.  That said, depending on the "log container page only" setting, we may only log URLs of a specified content type.  This feature is meant to reduce the number of logs that are generated (mostly images and other code that you may not find useful).  If, however, you do want everything logged, simply disable container page logging.  As mentioned in sdarapuneni's post, if you've enabled container page only, it will only log URLs of the specified content-type.  So for example, if you set container pages to only content type image/gif, then in your example above, you would only see log entries for the .gif files. 

[Sly_Cooper]

Is there any place where I can find the list of content types used? We are migrating from our existing proxy to PAN URL filtering and the current proxy does full logging. I want to make sure that I don't miss any logs which may be useful for investigation. Will there be any impact on device or Panorama if I disable "log container pages" only?

[dyang]

Hi Sly,

Thanks for your question.  You can see the default list of content types we use for container page logging by going to Device-->Setup-->Content-ID-->Content-ID Features-->Container Pages

If you would like to generate logs for other content types, then you can create your own container page profile and add the content-types that you'd like to log.  Please note that if no other profile is specified and you have "log container page only", then it will use the default profile.  If you've created your own container page profile, then we will use that instead. 

Alternatively, you can also uncheck the "log container page only", which means that you will log everything.  The impact here is obviously more logs, which could mean that you hit your log quota faster, and also potentially place a greater load on the device.  Given the number of page elements websites load these days, the recommendation is to add the content types of interest so that you're not flooded with logs.

Hope this helps,

Doris

[RNC]

I believe PANOS also logs URL each time you re-match against an app-id if a URL log profile is applied to the rule.

So if you match facebook (as an example), you may see the URL logged twice, as a log event is triggered for a the inital match of web-browsing. You may only see this in detail if log container pages is switched off.

My experience is to that you need to watch out for i/o loads on the device caused by URL logging. The rates can get out of hand easily, which has knock on effects on other sub-systems. The SDD based devices have better i/o, but if you intend to log up to Panorama, then keep that in mind too.