Session setup fails due to session hash collision error
65476
Created On 09/25/18 19:22 PM - Last Modified 09/15/20 16:59 PM
Symptom
Environment
- PAN-OS
- NAT
Cause
On a Palo Alto Networks firewall, a session is defined by two uni-directional flows each uniquely identified by a 6-tuple key: source-address, destination-address, source-port, destination-port, protocol, and security-zone.
A hash collision occurs when the firewall attempts to create a new session with either flow matching an existing session's 6-tuple key. When this occurs, the firewall will drop the incoming packet and increment the global counter called session_hash_insert_duplicate.
In the next two diagrams you can see this session collision behavior with two SIP calls. Each diagram includes a local phone, firewall, and a remote phone. Additionally we've enumerated the network legs of the client-server flow and the IP address and UDP port information for each packet.
- The local phone (192.168.1.2) initiates a SIP call to the remote phone (10.10.10.10) with a source and destination UDP port of 5060.
- The outbound traffic matches a source NAT policy on the firewall, translating the source IP to 10.10.10.1 and source port to 56611. The firewall then creates a session with two uni-directional flows. Here we can see the session details of the first SIP session created by the firewall.
> show session id 27954 Session 27954 c2s flow: source: 192.168.1.2 [l3-trust] dst: 10.10.10.10 proto: 17 sport: 5060 dport: 5060 state: ACTIVE type: FLOW src user: unknown dst user: unknown s2c flow: source: 10.10.10.10 [l3-untrust] dst: 10.10.10.1 proto: 17 sport: 5060 dport: 56611 state: ACTIVE type: FLOW src user: unknown dst user: unknown
- A second connection is attempted by the remote phone shortly afterward. This connection is destined for the firewall's IP address of 10.10.10.1. The firewall has an inbound destination NAT policy configured for the internal phone's IP address of 192.168.1.2. This call is never established, but why?
After running a filtered packet capture we see that the firewall is dropping the SIP INVITE packets from 10.10.10.10.
The global counters indicate that the drops occur during session creation and are caused by a session hash collision.
> show counter global filter delta yes packet-filter yes Global counters: Elapsed time since last sampling: 9.170 seconds name value rate severity category aspect description -------------------------------------------------------------------------------- session_allocated 1 0 info session resource Sessions allocated session_freed 1 0 info session resource Sessions freed session_install_error 1 0 warn session pktproc Sessions installation error session_install_error_s2c 1 0 warn session pktproc Sessions installation error s2c session_hash_insert_duplicate 1 0 warn session pktproc Session setup: hash insert failure due to duplicate entry
The firewall matches the call from 10.10.10.10 with an inbound destination NAT policy and translates the destination IP address from 10.10.10.1 to 192.168.1.2. When the firewall attempts to create a session it checks to see if the session hash matches an existing c2s or s2c flow.
If we compare the c2s flow of the first session and the expected s2c flow of the failed session we have the following:
| c2s flow: source: 192.168.1.2 [l3-trust] dst: 10.10.10.10 proto: 17 sport: 5060 dport: 5060 state: ACTIVE type: FLOW src user: unknown dst user: unknown | s2c flow: source: 192.168.1.2 [l3-trust] dst: 10.10.10.10 proto: 17 sport: 5060 dport: 5060 state: ACTIVE type: FLOW src user: unknown dst user: unknown |
Notice that these two flows are identical. If the firewall had created the second session, it would be unable to identify which flow to use for a matching packet. The firewall does not create the second session due to this conflict.
Resolution
This behavior can manifest itself in unique ways depending on the implementation. For this scenario, a user may reconfigure their source NAT from Dynamic IP and Port to Static NAT. This will allow the second call to match the initial s2c flow instead of creating a new session by the firewall.
When session_hash_insert_duplicate is seen in the global counters, keep in mind how NAT policies could affect new sessions that match existing flows.