server hello message dropped at firewall

[Kanthanathan]

We are facing currently this issue with a DC firewall. The following is the environment

EnduserPC-> DC Firewall (PAN) -> f5 Load Balancer-> Web Servers

All these days the users were able to login to the web services without any hassles. For the last 2 days, we found that the users are not able to access the application properly. f5 load balancer is showing lot of ssl handshake error. 

We raised a support ticket with f5 and identified that the client is sending client hello and f5 is responding with server hello. However the server hello message is missing at the client side. The client keeps retransmitting packet and after about 10 sec the ssl session timeout happens. 

While we did a packet capture at the firewall we noticed that the firewall indeed receive the server hello message. We are not sure why the firewall is dropping the server hello message. we have disabled all security inspection at the firewall including ssl decryption. 

Firewall is running PAN OS 9.0.13 and we also tried upgrading this to 9.1.10. But still the issue persists. 

Can someone guide us what could be the issue. 

[dmifsud]

When you say the firewall is dropping this, did you see the server hello in the 'drop' stage capture? Or did you see it in the receive stage capture, but not the transmit?

If you see it in the drop stage, you can usually get more info on the drop reason by checking the global counters:

- When your pcap filter is set and enabled, log onto the CLI

- Run the command 'show counter global filter packet-filter yes delta yes'

- Test the connection, allow it to fail

- Run the command 'show counter global filter packet-filter yes delta yes' - this time check for any 'drop' counters

It's important that you also check the pcap at the 'transmit' stage - if you see the packet in the transmit stage, it is not dropped by the firewall. 99% of the time when I see and issue with server hello dropping somewhere, it's because of MTU as it's the first part of the handshake which will send a full sized packet.

Basically, make sure you pcap at receive, transmit and drop. Receive only is not enough for a diagnosis.

- DM

[Kanthanathan]

Hi,

Thank you for the response. 

All of a sudden without any change to configuration or environment change, the app started working through the firewall for the last 8 hours. The service has been down for more than 50 hours, Since it is a long week end we will not be able to observe this in the next 3 days. We will check pcap at all the stages if the problem repeats again and then revert back here. 

Regards, 

Kanthan