Unknown IP Rate Limit Mitigation for User-ID Mappings

• Users are not identified and appear as 'unknown' in the firewall’s user-IP mappings. This can result in users matching a wrong rule and cause traffic to be dropped or blocked.
• The following message appears in the User-ID log (useridd.log):

> tail follow yes mp-log useridd.log

2014-01-20 14:07:45.498 +0100 pan_user_id_agent_update_unknown_ip_rate_limit: Unknown IP rate is now 101, enabling rate limiting for VM1_collector

• A check on the number of unknown IP-user mappings returns a high value, for example: 

> show user ip-user-mapping all type UNKNOWN option count

Total: 349 users

• User-ID IP Mapping
• All PAN-OS Versions

Service devices that are neither a component of the corporate domain nor have a real user behind them are often overlooked when designing the User-ID topology. These devices, 'bring your own device' (BYOD) and smart phones, behave as employee workstations and generate sessions to the corporate firewall, creating additional stress on the User-ID topology. Smart phones double the number of users that need to be identified and matched to the correct security policy. Because of this scenario, it is possible to reach the limit for unknown IP addresses and User-ID agents probed on the Palo Alto Networks firewall.

This process happens when the rate of sending queries for unknown IP addresses from the firewall to the User-ID Agent becomes over 100 unknown IP addresses per second. In the time when this log is generated, the firewall has many unknown IP addresses in the user-IP mappings. At this point, if any request for an unknown IP address needs to be sent to the User-ID Agent, the query is dropped and the mapping is not requested from the User-ID Agent.

Note: The limit on the firewall is 100 requests for unknown IP addresses per second, which is a high rate even for the largest implementations. Most users, with even the default settings, likely don't notice this issue during the lifespan of the firewall.

The rate limiting normally lasts for a couple of seconds (depending on the network) and the administrator can see the rate limiting being removed in the same log file (useridd.log):

2014-01-20 14:07:45.498 +0100 pan_user_id_agent_update_unknown_ip_rate_limit: Unknown IP rate is now 101, enabling rate limiting for VM1_collector

2014-01-20 14:07:47.504 +0100 pan_user_id_agent_update_unknown_ip_rate_limit: Unknown IP rate rate is now 76, disable rate limiting for VM1_collector

Running the same operational command after the rate-limiting is finished results in a significantly smaller number of unknown users. For example:

> show user ip-user-mapping all type UNKNOWN option count

Total: 21 users

During that period (2 seconds in the above example), all users that needed IP-user mapping were discarded. The rate-limiting period is small and the “unknown” ip-user-mappings have significantly smaller expiration timers than the identified users. However, it is possible that users are matched to wrong rules due to this limiting process. This will be accompanied by unusual traffic logs, where a user is mapped as a source of traffic, followed by an “unknown” user as a source from the same IP address, then later followed by the user being correctly mapped again. The process depends on the activity of the user. If the user initiates another session, then a new request is sent to the User-ID Agent because the firewall has “unknown” user assigned to that IP address. The agent replies with the mapping.

The high rate of unknown user-IP requests from the firewall can occur when there are many systems that don't have users behind them. They include IP phones, mobile phones, printers, wireless access points, servers and workstations that are not part of the domain, and other machines used in the corporation. Since these devices are frequently initiating sessions and don't have users behind them, the Palo Alto Networks firewall is constantly trying to map them.

See below for some FAQ's and more information about this issue:

• What is the actual impact of Userid enabling rate-limiting for unknown IPs?  
  1. The PA is rate-limiting the send requests for ip-user-mapping because there are more than 100 unknown-users per second generating traffic that was hitting the DP.
  2. The rate-limiting period is small and the “unknown” ip-user-mappings have significantly smaller expiration timers than the identified users. However, it is possible that users are matched to wrong rules due to this limiting process. This will be accompanied by unusual traffic logs, where a user is mapped as a source of traffic, followed by an “unknown” user as a source from the same IP address, and then later followed by the user being correctly mapped again. The process depends on the activity of the user. If the user initiates another session, then a new request is sent to the User-ID Agent because the firewall has “unknown” user assigned to that IP address. The agent will reply with the mapping.
• Are there any mitigations available besides zone include/exclude ACLs that we should know about? 
  1. Global on Firewall
     1. You may want to use the feature below but this only implies to the user-ip mapping that you would like to learn.
        • Define Subnetworks to Include/Exclude for User Mapping
        • Device > User Identification > User Mapping
        • Use the Include/Exclude Networks list to define the subnetworks that the User-ID agent will include or exclude when performing IP address-to-username mapping (discovery).
     2. Per Zone on Firewall w/UserID enabled on it:
        1. See documentation in current article
        2. BIG CAVEAT - From the Online Help:
           • "If you add entries to the Exclude List but not the Include List, the firewall excludes user mapping information for all subnetworks within the zone, not just the subnetworks you added."
           • We are including the RFC1918 networks in the Include List since the excludes have precedence over the includes.

• Is there a "nerd knob" that can be adjusted on larger platforms to raise the threshold? 
  1. No, the Rate is hardcoded to 100 unknown-users per second regardless of platform