Palo VM firewall drop packets behind Azure load balancer

[Vanessaxu]

The topoplogy is

spoke subnet ---> Aure LB ---> 2x Palo VM firewalls -> express route --> on-prem Palo firewall --> on-prem server

user at spok subnet send files to onprem is very slow. we did iperf test from a subnet in the spoke vnet to an onprem test server. There are drops on both of the firewalls that behind the LB. The dropped packets are normal tcp ack, fin-ack, rst ack cwr, and tcp retrsnmission.

we did another iperf test from a different subnet in the same spoke vnet and skip the Azure LB , just go through one of the Palo vm firewall. Then there is no drops on this Palo firewall.

also, there is no drop on the on-prem palo firewall.

what could cause the drop on the palo vm firewalls when behind the Azure LB? could anyone help? Thank you!

Cloud NGFW for Azure 

[BPry · Accepted solution]

@Vanessaxu,

How did you have iperf setup when you were doing your testing? If you didn't maintain the same source port and destination port in your testing then you'd expect it to split the traffic across both PA-VMs due to the Azure LB utilizing 5-tuple hashing by default. Generally speaking 5-tuple works perfectly fine for most operations and helps split the load as much as possible.

It's possible in your scenario that you would want to utilize 2-tuple or 3-tuple session persistence depending on how you're transferring the file.

[Vanessaxu]

iperf dose uses different source port and destination port is same.  we uses the 5-tuple hashing as the rule set to none.  

I will test to use 2 tuple or 3 tuple session persistence.  but why the default 5-tuple hashing will cause the firewall drop packets?

Thank you!

[Vanessaxu]

we changed to 3-tuple session persistence and no more packet drop.  

Thank you for your suggestion!