How to Enable Increased HTTP Header Logging

Overview

This document describes how to create a custom URL Content Type whenever the Wildfire Submission log entry detail does not display a value for User-Agent.

With the release of PAN-OS 6.1, HTTP header fields have been added for fields that assist with forensics and troubleshooting tasks. The primary use cases are for threat analysis, such as the following:

• A user is alerted of malicious C2 traffic in the threat log from a client behind a web proxy and wants to identify the infected client. The XFF field in the threat log can be used to identify the infected client.
• A user may be actively investigating the traffic and URL logs of a compromised host. Reviewing the user-agent strings used by the client can help to identify illegitimate user agents or possibly data exfiltration (or data exrusion), the unauthorized transfer of data from a computer.
• Malware is identified during the investigation of the traffic and URL logs entries associated with a compromised host a malicious drive-by page used to serve up. What is needed, however, is the page performing the redirect such as the referrer to block the referrer, alert the referrer owner, or find other possibly compromised hosts, etc.
• There is a WildFire report for a malicious file, which makes it necessary to identify the infected host behind a web proxy. The X-Forward-For value in the WildFire report can identify the host.

The logging of these fields is visible in the Detailed Log view of a session in the URL Filtering, Threat, and Wildfire Submission logs. When viewing in the Detailed Log view, it will display related logs at the bottom. When selecting one of the related logs, the contents of the view changes. This allows one to view the details between related logs without having to leave this dialogue window. For example, a URL Filtering log entry reveals the URL from which a file was retrieved. The detail of this log may show the Wildfire Submission log associated with the forwarding of the retrieved file to Wildfire for analysis. The detail of the Wildfire Submission log can now display HTTP header information associated with this event.

Details

Logging HTTP Header Information

Before PAN-OS 6.1

Prior to the release of PAN-OS 6.1, users could choose to use the Source User column in the URL Filtering log to display the X-Forwarded-For value. This would be enabled in Device > Setup > Content-ID.

Once the gear to edit the properties is selected, check the x-forwarded-for box to display the contents of this field in the Source User column.

This setting is NOT a prerequisite for, nor is it associated with, the PAN-OS 6.1 HTTP Header logging feature and should be considered separately. In PAN-OS 6.1 and later, it may be considered to be a legacy setting.

PAN-OS 6.1 and Later

The URL Filtering log now has a separate column for the X-FORWARDED-FOR value from the HTTP header.

Note: The HTTP header logging is enabled on the Settings tab in a URL Filtering Profile.

The HTTP Header logging is visible in the log detail of a Wildfire Submissions log entry.

In order for this detail to show up in the Wildfire log, remove the check in the "Log container page only" in the URL Filtering security profile. When this box is checked only the main page that matches a URL category is logged, not any subsequent pages that may be loaded within the main page. Unchecking this box will populate the log detail HTTP Header fields, but the amount of logging could overwhelm the firewall. Instead, it is recommended to create a custom URL Content Type.

Adding HTTP Header Columns to the URL Filtering Log

1. In Monitor > URL Filtering, add the new columns to the table by highlighting a column next to where the new ones will be placed.
2. Select the downward triangle to expose the contextual menu.
3. Slide over Columns to display the list of columns. The order of the column choices in the list will vary.

It may be useful to have URL, X-Forwarded-For, Referrer, and/or User-Agent displayed as column headers. This is how it will now look:

Selecting the magnifying glass icon in the left-most column of the log table will open the detailed log view window, where there is a section for the HTTP headers

Notice in the HTTP Headers section in the image not all of the fields will appear filled out with information. Sometimes, the application that is making the network connection is not known to PAN-OS.

Defining A Custom URL Content Type

If "Log Container Pages Only" is disabled, the amount of logging on the firewall increases, which can lead to undesirable effects. Alternatively, the HTTP Header Fields section can be populated in the detailed log view of a Wildfire URL Submission entry by creating a custom URL content type.

1. First, identify the content type. One good way to do this is create a packet capture of a session.
   
   In the picture above, the Content-type associated with the GET of /public/api/test/pe is "application/octet-stream".
2. Go to Device > Setup > Content ID > Container Pages.
3. Click Add to create a new container page object.
   
   Clicking Add will open the Custom URL Content Type window below.
4. Select Add again to enter "application/octet-stream".
   
5. When creating a Custom URL Content Type object, include the predefined objects.
6. Select Add multiple times to add them all. This results in predefined objects in addition to the ones added.
   

Repeat this process whenever the Wildfire Submission log entry detail does not display a value for User-Agent.

owner: jjosephs