Symantec WebFilter to PAN-DB URL Filtering Migration Guide

sshibiraj · ‎01-24-2019

The second part of the document contains examples on how to migrate from Symantec Web Filter categories to PAN-DB categories and how to use them in the security policies of the next-generation firewall. During the migration, it is a recommended best practice to configure a URL Filtering profile with all categories set to “alert” in parallel with your web filtering solution. This allows you to run reports in PAN-OS and Proxy-SG to verify policies and category mappings before switching the URL filtering functions over completely to PAN-DB URL categorization on our next generation firewall.

The third part of this document contains usage examples and recommended security best practices when using PAN-DB for URL categorization in the next-generation firewall.

Category Mappings for Moving From Symantec WebFilter to PAN-DB

To start the migration, the first thing we recommend is to review the categories that are blocked by policy with the Symantec WebFilter and map them to the corresponding PAN-DB URL categories.

The Symantec WebFilter Database is organized into 85 URL categories. You can find a complete list and definitions of the categories at this link: https://sitereview.bluecoat.com/category-descriptions.

PAN-DB is organized into more than 65 URL categories. You can find a complete list and definitions of the categories at this link: Complete List of PAN-DB URL Filtering Categories.

Symantec WebFilter offers a service called “Site Review.” The purpose of “Site Review” is to allow Symantec customers to check the current database categorization of WebFilter URLs and report sites that they believe are incorrectly categorized. You can find the "Site Review" at this link: https://sitereview.bluecoat.com/.

PAN-DB URL filter also offers a service called “Test a Site.” The purpose of “Test a Site” is to allow Palo Alto Networks customers to check the current database categorization of PAN-DB URLs and report sites that they believe are incorrectly categorized. You can find "Test a Site" at this link: https://urlfiltering.paloaltonetworks.com/.

The table below will help you with the category mapping exercise.

Symantec	Palo Alto Networks	Differences	Recommendations
Abortion	Abortion
Adult/Mature Content	Adult or Questionable
Alcohol	Alcohol and Tobacco
Alternative Spirituality/Belief	Religion
Art/Culture	Entertainment and Arts
Auctions	Auctions
Audio/Video Clips	Streaming Media or Music
Brokerage/Trading	Stock Advice and Tools or Financial Services
Business/Economy	Business and Economy
Charitable Organizations	Society
Chat (IM)/SMS	Internet Communications and Telephony
Child Pornography	Adult
Computer/Information Security	Computer and Internet Info or Hacking
Content Servers	Content Delivery Networks
Controlled Substances	Abused Drugs
Dynamic DNS Host	Dynamic-DNS		Best Practice recommendation, Block “dynamic-dns” category
E-Card/Invitations	Shareware-and-Freeware
Education	Educational Institutions
Email	Web-based Email
Entertainment	Entertainment and Arts
Extreme	Extremism		Best Practice recommendation, Block “extremism” category
File Storage/Sharing	Online Storage and Backup
Financial Services	Financial Services
For Kids	Society	This Symantec category is not a stand-alone category
Gambling	Gambling
Games	Games
Government/Legal	Government
Hacking	Hacking
Health	Health and Medicine
Humor/Jokes	Entertainment and Arts or Questionable
Informational	N/A	This Symantec category is not a stand-alone category	Recommended action: – Use “Test a Site” to find corresponding PAN-DB category for matching websites – Or Create a Custom URL category and control matching websites
Internet Connected Devices	Computer and Internet Info	There is no one-to-one mapping for this category. This is a subset of “computer-and-internet-info” category	Recommended action: – Use “Test a Site” to find corresponding PAN-DB category for matching websites – Or Create a Custom URL category and control matching websites
Internet Telephony	Internet Communications and Telephony
Intimate Apparel/Swimsuit	Swimsuits and Intimate Apparel
Job Search/Careers	Job Search
Malicious Outbound Data/Botnets	Command-and-Control		Best Practice recommendation, Block “Command-and-Control” category
Malicious Sources/Malnets	Malware		Best Practice recommendation, Block “malware” category
Marijuana	Abused Drugs
Media Sharing	Streaming Media or Online Storage and Backup
Military	Military
Mixed Content/ Potentially Adult	Adult, Nudity or Questionable	Based on the category description provided by Symantec, most URLs should be mapped to “adult” but the URLs could also be part of “nudity” or “questionable”
News/Media	News
Newsgroups/Forums	News or Personal-Sites-And-Blogs
Non-Viewable/Infrastructure	Insufficient Content
Nudity	Nudity
Office/Business Applications	Computer and Internet Info
Online Meetings	Internet Communications and Telephony
Peer-to-Peer (P2P)	Peer-to-Peer
Personals/Dating	Dating
Personal Sites	Personal Sites and Blogs
Phishing	Phishing		Best Practice recommendation, Block “phishing” category
Piracy/Copyright Concerns	Copyright-Infringement		Best Practice recommendation, Block “copyright-infringement“ category
Placeholders	Parked		Best Practice recommendation, Block “parked” category
Political/Social Advocacy	Philosophy and Political Advocacy
Pornography	Adult
Potentially Unwanted Software	Shareware and Freeware or Questionable
Proxy Avoidance	Proxy Avoidance and Anonymizers		Best Practice recommendation, Block “proxy-avoidance-and-anonymizers” category
Radio/Audio Streams	Streaming Media
Real Estate	Real Estate
Reference	Reference and Research
Religion	Religion
Remote Access Tools	Internet Communications and Telephony
Restaurants/Dining/Food	Society
Scam/Questionable/Illegal	Questionable
Search Engines/Portals	Search Engines
Sex Education	Sex Education
Sexual Expression	Adult or Society	If the website content pertains to sexual identity then the category will be "society" If not, the category will be "adult"
Shopping	Shopping
Social Networking	Social Networking
Society/Daily Living	Society
Software Downloads	Shareware and Freeware or Computer and Internet Info
Spam	Questionable	URLs related to spam are included in the category "questionable" This category also includes websites with illegal, immoral and offensive content
Sports/Recreation	Sports
Suspicious	Insufficient Content or Questionable
Technology/Internet	Computer and Internet Info
Tobacco	Alcohol and Tobacco
Translation	Translation
Travel	Travel
TV/Video Streams	Streaming Media
Uncategorized	Unknown		Best Practice recommendation, Block “unknown” category
Vehicles	Motor Vehicles
Violence/Hate/Racism	Extremism		Best Practice recommendation, Block “extremism” category
Weapons	Weapons
Web Ads/Analytics	Web Advertisements
Web Hosting	Web Hosting

Roll Out URL Category Enforcement

The recommended practice for deploying URL filtering in your organization is to start with a “passive" URL Filtering profile that will create log entries by employing the “alert” policy action on all categories in parallel with your existing web filter appliance.
- On the Palo Alto Networks firewall, create a new URL Filtering profile.
  - Select Objects > Security Profiles > URL Filtering.
  - Select the default profile and then click Clone. The new profile will be named default-1.
  - Select the default-1 profile and rename it. For example, rename it to URL-Monitoring.
- Configure the action for all categories to alert.
  - In the section that lists all URL categories, select all categories.
  - To the right of the Action column heading, mouse over and select the down arrow and then select Set Selected Actions and choose Alert.
  - Click OK to save the profile.
After setting the Alert action, you can monitor user web activity through URL filtering reports on both appliances for a few days or weeks to determine accuracy of the provided category mappings. Palo Alto Networks recommends to validate accuracy for the top 1,000 websites seen by your organization.
- Apply the URL Filtering profile to the security policy rule(s) that allows web traffic of users.
  - Select Policies > Security and select the appropriate security policy to modify it.
  - Select the Actions tab and in the Profile Setting section, click the drop-down for URL filtering and select the new profile.
  - Click OK to save.
- View the URL filtering logs to determine all of the website categories that your users are accessing.
- For information on viewing the logs and generating reports, see Monitor Web Activity.
  - Select Monitor > Logs > URL Filtering. A log entry will be created for any website that exists in the URL filtering database in a category that is set to any action other than allow.
In this procedure, all categories will be set to alert, which will cause traffic for all websites to be logged. This may potentially create a large amount of log files, so it is best to do this for initial monitoring purposes to determine the types of websites your users are accessing and compare URL categories triggered.

Collect all URL category objects used in the Symantec Web Filter Policy Manager and map them into PAN-DB URL categories using the provided URL category map after vetting them for accuracy using the above steps.

After determining the categories that your organization allows users to access, set the policy action to “allow” for these URL categories on the next-generation firewall. The firewall does not generate logs for traffic matching these URL categories.

You can then make decisions on the URL categories that should be controlled according to the company policy by setting the appropriate policy action to each of these categories in the URL Filtering profile(s). The recommended actions column of the "URL category" table in the previous section and the "best practices" section at the end of this document are provided to further assist you in making policy decisions.

If possible, it is recommended to use a “slow roll” approach, using USER-ID as described below when deploying these newly created URL Filtering profile(s) to Security policies.
- Clone an existing policy that allows web access and add an additional match criteria on User set to a single department. (i.e., IT, Marketing, Engineering, etc.)
- Add the new URL Filtering profile to this Security policy and move the policy above all policies that allow web access since policy rules are matched top down.
- Monitor the above policy for usage and get feedback from the users belonging to the Group Object.
- Incorporate changes as necessary to the URL Filtering profile before adding it to all other applicable security policies.

Best Practice URL Filtering Profile

Attach a URL Filtering profile to all rules that allow access to web-based applications to protect against URLs that have been observed hosting malware or exploitive content.
As a best practice, use PAN-DB URL filtering to prevent access to web content that is at high-risk for being malicious.
These include command-and-control, copyright-infringement, dynamic-dns, extremism, malware, phishing, proxy-avoidance-and-anonymizers, unknown, and parked. The best practice URL Filtering profile sets all known dangerous URL categories to block.
Failure to block these dangerous categories puts you at risk for exploit infiltration, malware download, command and control activity, and data exfiltration.
In addition to blocking known bad categories, you should also alert on all other categories so you have visibility into the sites your users are visiting.
If you need to phase in a block policy, set categories to continue and create a custom response page to educate users on your acceptable use policies and alert them to the fact that they are visiting a site that may pose a threat.
This will pave the way for you to outright block the categories after a monitoring period.

What if I can’t block all of the recommended categories?

If you find that users need access to sites in the blocked categories, consider creating an allow list for just the specific sites if you feel the risk is justified. On categories you decide to allow, make sure you set up credential phishing prevention to ensure that users aren’t submitting their corporate credentials to a site that may be hosting a phishing attack. Allowing traffic to a recommended block category poses the following risks:

Malware – Sites known to host malware or used for command and control (C2) traffic. May also exhibit Exploit Kits.

Phishing – Known to host credential phishing pages or phishing for personal identification.

Dynamic-dns – Hosts and domain names for systems with dynamically assigned IP addresses and which are oftentimes used to deliver malware payloads or C2 traffic. Also, dynamic DNS domains do not go through the same vetting process as domains that are registered by a reputable domain registration company and are, therefore, less trustworthy.

Unknown – Sites that have not yet been identified by PAN-DB, perhaps because they were just registered. However, these are sites that are oftentimes generated by domain generation algorithms and are later found to exhibit malicious behavior.

Command-and-control – Command-and-control URLs and domains used by malware and/or compromised systems to surreptitiously communicate with an attacker's remote server to receive malicious commands or exfiltrate data.

Copyright-infringement – Domains with illegal content, such as content that allows illegal download of software or other intellectual property. This category was introduced to enable adherence to child protection laws required in the education industry as well as laws in countries that require internet providers to prevent users from sharing copyrighted material through their service.

Extremism – Websites promoting terrorism, racism, fascism or other extremist views discriminating people or groups of different ethnic backgrounds, religions, or other beliefs. This category was introduced to enable adherence to child protection laws required in the education industry.

Proxy-avoidance-and-anonymizers – URLs and services often used to bypass content filtering products.

Parked – Domains registered by individuals, oftentimes later found to be used for credential phishing. These domains may be similar to legitimate domains. For example, pal0alto0netw0rks.com, with the intent of phishing for credentials or personal identify information. Or, they may be domains that an individual purchases rights in hopes that it may be valuable someday, such as panw.net.

USE URL Categories to Define Traffic to Decrypt or Not Decrypt

Plan to decrypt as much traffic that is not private or sensitive as your firewall resources allow to reduce the attack surface by exposing and preventing encrypted threats. Understand local laws and regulations about the traffic you can legally decrypt and user notification requirements.

Please see documentation for SSL Decryption deployment and pre-requisites. The below steps describe Decryption policy definitions only.

Create a “no-decrypt” policy that will prevent any website mapping to the recommended "no-decrypt" URL category list (i.e., financial-services, health-and-medicine, government, etc.) from being decrypted.

Even while not using decryption it is a recommended best practice to create a Decryption Profile to block sessions with expired certificates or untrusted issuers and use it with your "no-decrypt" policy.
Navigate to Objects > Decryption Profile.
Add a Profile called “nodecrypt” and check “Block sessions with expired certificates” and “Block sessions with untrusted issuers” under No Decryption tab.
Navigate to Policies > Decryption and click Add.
Enter a Name and optionally enter a Description and Tag(s).
On the Source tab, enter the zone where the users are connected.
On the Destination tab, enter the zone that is connected to the internet.
On the URL Category tab, click Add and select the financial-services, government, and health-and-medicine URL categories.
On the Options tab, set the action to No Decrypt.
Also set Decryption Profile to a “nodecrypt”.
Click OK to save the policy rule.

Create a “must-decrypt” policy that will decrypt any website mapping to the recommended "must-decrypt" URL category list.
(Malware, Phishing, Unknown, Command-and-control, Copyright-infringement, Proxy-avoidance-and-anonymizers, Content-deliver-networks, Parked, Web-based-email, Social Networking, Personal-sites-blogs, Web-hosting, Insufficient-content, Not-resolved, Online-storage and backup, Hacking, Questionable, Dynamic DNS)

Navigate to Policies > Decryption and click Add.
Enter a Name and optionally enter a Description and Tag(s).
On the Source tab, enter the zone where the users are connected.
On the Destination tab, enter the zone that is connected to the internet.
On the Service/URL Category tab, enter all the Recommended URL Categories
On the Options tab, set the Action to Decrypt and the Type to SSL Forward Proxy.
Use a Decryption Profile along with your decryption policy to block sessions that fail on SSL decryption.
Ensure that this "must-decrypt" policy is listed after the "no-decrypt" policy to ensure that rule processing occurs in the correct order.

Create a “best-effort-decrypt” policy that will decrypt all other traffic using the same steps as above but with URL categories set to "any" and a Decryption Profile with options under Failure Checks section unchecked. This ensures that sessions are allowed even if SSL decryption fails.

With these three decrypt policies in place, any traffic destined for the financial-services or health-and-medicine or government URL categories will not be decrypted. All other traffic will be decrypted.

URL Filtering Use Cases

Case-1: Policy to block download of high-risk file types from certain categories [Decryption + URL Filtering + File-Blocking + Threat Prevention]

Please refer to SSL Decryption Best Practices to enable SSL decryption. This is necessary to accurately inspect, classify, and block encrypted traffic.
Create a File Blocking Security Profile.

The default basic file blocking profile can be used, or we can create a custom profile based on the basic file blocking profile.
Select Objects > Security Profiles > File Blocking.
Select “basic file blocking” or “strict file blocking” profile and click Clone.
Rename the profile and edit to select appropriate file types for Block, Continue, and Alert actions based on your company policy or use the default profile provided.
Click OK.

Create the security policy rule that will block risky file downloads from specific categories.

This rule must precede other rules because it is a specific rule. (More specific rules must precede other rules.)
Select Policies > Security and click Add.
Enter a Name and optionally a Description and Tag(s).
On the Source tab add the zone where the users are connected.
On the Destination tab, select the zone that is connected to the internet.
On the Service/URL Category tab, Add the specific categories from which risky file download needs to be blocked. (i.e., Web-hosting, Personal-sites-blogs, Social Networking, Peer-to-Peer, Online-storage and backup, Web-based-email, Copyright-infringement, Shareware-and-freeware).
On the Actions tab, select Action “Allow” and add the default profiles for Antivirus, Vulnerability Protection, Anti-Spyware, URL Filtering, and the newly created File Blocking profile.
Click OK to save the security profile.
Commit the Configuration.

With this Security policy rule in place, any user trying to download executable files or other risky files from Dropbox, Box, or any free software download websites will be blocked.
Because this rule will also allow access to the internet, threat prevention profiles are applied to the rule, so traffic that matches the policy will be scanned for threats. This is important because the allow rule is terminal and will not continue to check other rules if there is a traffic match.

Case-2: Policy to Control Web Access [Decryption + User-ID + App-ID + URL Filtering + Data Filtering + Threat Prevention]

In this use case, users belonging to the Marketing group, for example, have access to Box for collaboration but not to any of the other “online-storage-and backup” vendors. All other users are blocked from all “online-storage-and-backup” applications. The company policy also states that documents marked “Confidential” should not be shared on Box by the Marketing group.

Please refer to the SSL Decryption Best Practices to enable SSL decryption. This is necessary to accurately inspect, classify, and block encrypted traffic.
Create a Security Policy that will block all users from accessing “online-backup-and-storage” applications. This can be done either using a specific Security Policy or as part of a URL Filtering profile that would be included in all Security Policies that allow internet access.

Select Policies > Security and click Add.
Enter a Name and optionally a Description and Tag(s).
On the Source tab add the zone where the users are connected.
On the Destination tab, select the zone that is connected to the internet.
On the Service/URL Category tab, click Add and add the online-storage-and-backup category.
On the Actions tab, select Action Deny.
Click OK to save the security profile.

Create a Data Pattern Custom Object and add it to a data filtering Security Profile.

Select Objects > Custom Objects > Data Patterns and click Add.
Select Pattern Type as “Regular Expression”.
Select File Type as “Any”.
Set Data Pattern to “.*((Confidential)|(CONFIDENTIAL))”.
Click OK.
Select Objects > Security Profiles > Data Filtering and click Add.
Set the Data Pattern Field to the above created object.
Set Alert/Block Threshold to 1 and Log Severity to Critical and Click OK.

Create the Security Policy that will allow the Marketing group to access Box application. Because this allow rule will also allow access to the internet, threat prevention profiles are applied to the rule, so traffic that matches the policy will be scanned for threats.

This rule must precede other rules because it is more specific than the other policies.
Select Policies > Security and click Add.
Enter a Name and optionally a Description and Tag(s).
On the Source tab add the zone where the users are connected.
On the User tab in the Source User section click Add.
Select the directory group that contains your marketing users.
On the Destination tab, select the zone that is connected to the internet.
On the Applications tab, click Add and add the boxnet App-ID signature.
On the Actions tab, add the default profiles for Antivirus, Vulnerability Protection, and Anti-Spyware.
Also add the Data Filtering profile that was created in the previous step.
Click OK to save the Security Profile and commit the configuration.

With these policies in place, any user who is part of the Marketing group will have full access to Box application and any user that is not part of the Marketing group will be blocked from all online-storage-and-backup websites.
Additionally, all files that are shared on Box will be scanned for the keyword “Confidential” and blocked if found. An entry will also be logged under Monitor > Logs > Data Filtering.

Case-3: Subscribe to an external malicious URL feed [URL Filtering + External Dynamic Lists]

In this use case, administrator wants the firewall to ingest an external feed that provides IOCs (Indicators of Compromise) in the form of URLs. This dynamic list of URLs has to be continuously updated in policy and blocked by Palo Alto Networks next generation firewall without any manual intervention.

To protect your network from new sources of threat or malware, you can use External Dynamic List in URL Filtering profiles to block or allow or to define granular actions such as continue, alert, or override for URLs before you attach the profile to a Security Policy rule. Unlike the allow list, block list, or a custom URL category on the firewall, an external dynamic list gives you the ability to update the list without a configuration change or commit on the firewall.

Navigate to Objects > External Dynamic Lists.
Click Add.
Select Type “URL List”.
Enter Source [this could be a web server hosting a file of URL’s].
Select appropriate Frequency of checks using the Repeat field.
Click OK.

Navigate to Objects > Security Profiles > URL Filtering.
Select appropriate URL Filtering profile.
The above created EDL should be seen as a custom category.
Assign appropriate policy action to this category.
This URL Filtering profile can now be added to a Security Policy(s).

With this Security Policy in place, any user attempting to connect to websites part of the URL feed will be blocked. This URL list is dynamically updated by the firewall without any commit required by the administrator. Any attempt to connect to these URLs is also logged under Monitor > Logs > URL Filtering.

Useful Resources

AustinClark · ‎04-10-2023

Is there an updated version to this?

Symantec WebFilter to PAN-DB URL Filtering Migration Guide