HTTPS traffic suddenly blocked

[MMCiobanu]

Hi,

We have had same issue twice in two days, where the firewalls would suddenly block HTTPS traffic; this happened on two platforms, PA-3020 and PA-5020, both running 5.0.8 PAN-OS, and the work around was to create a "Do-not decrypt all" decryption policy at the top, until we could schedule a reboot; the reboot seems to fixed the issue for now in both cases, but we are worried that  this will happen again.

Did anybody else experienced this issue or have an idea of what is happening?

Thank you

[HULK]

Hello,

Would it be possible for you to take a TCP FLOW-BASIC and CTD BASIC, while the problem occurred.

> Verify if there is an an session exist for the traffic on the firewall. you may use filters like ( addr.src in IP_ADD_OF_THE_TESTING_PC ) and ( addr.dst in IP_ADD_OF_THE_DESTINATION ) to check the security policy that the traffic hitting. Also you can check the real time session in the CLI by using 'show session all filter source IP_ADD_OF_THE_TESTING_PC destination IP_ADD_OF_THE_DESTINATION'.

>  If there is a session exist for the same traffic,  then please apply  CLI command PAN> show session id XYZ   >>>>>>>> to get detailed information about that session, i.e NAT rule, security rule, ingress/egress interface etc.

>  verify the global counters, if a specific "DRP" / "DECRYPTION" counter is increasing rapidly. The command show counter global provides information about the processes/actions taken on the packets going through the device; if they are dropped, nat-ed, decrypted etc.  These counters are for all the traffic going through the device and are useful in troubleshooting issues; like poor performance, packet loss, latency etc. It is advised to use the command show counter global filter packet-filter yes delta yes in conjunction with filters to obtain meaningful data.

For more information, you can follow the DOC What is the Significance of Global Counters?

> You can enable FLOW BASIC feature to understand the exact reason behind the failure:

> debug dataplane packet-diag clear all

> debug dataplane packet-diag set filter match source  IP_ADD_OF_THE_TESTING_PC destination IP_ADD_OF_THE_DESTINATION

> debug dataplane packet-diag set filter match source IP_ADD_OF_THE_DESTINATION destination  IP_ADD_OF_THE_TESTING_PC

> debug dataplane packet-diag set log feature flow basic / & ssl basic / & proxy basic

> debug dataplane packet-diag set log feature tcp all

> debug dataplane packet-diag set filter on

> debug dataplane packet-diag set log on

~~~~~~~~~~~~~~~~ Initiate traffic through the PAN firewall/try to browse a website HTTPS ~~~~~~~~~~~~~~~~~~~~~~~~~

> debug dataplane packet-diag set log off

> debug dataplane packet-diag aggregate-logs

> less mp-log pan_packetdiag_log.log

For more information, you can follow the DOC: Packet Capture, Debug Flow-basic and Counter Commands

Hope this helps.

[MMCiobanu · Accepted solution]

Hi,

Thank you very much for the answer; in mean while, I got somebody from PaloAlto Networks support team to take a look and he found that we were actually running out of SSL decrypt session buffer. He particularly looked at the Proxy session values, when running this command:

debug dataplane pool statistics

and at the total number of ssl-decrypt sessions, by running this command:

show session all filter ssl-decrypt yes count yes

We moved our mailboxes to the cloud, on Office365, and there were about 7000 ssl-decrypt session only related to this traffic alone. we found it using these commands:

show session all filter ssl-decrypt yes application ms-exchange count yes

show session all filter ssl-decrypt yes application rpc-over-http count yes.

We had to exclude all the internal clients going to Office365 servers from being decrypted, by adding a Decrypt rule:

This helped us drop down the ssl-decrypt sessions count; however, I am surprised that a 5Gbps capable firewall is not built to handle more than 16,000 concurrent ssl-decrypt session when almost all web traffic is now running over ssl.