topic Re: Web page issues between F5 and PA in General Topics

Web page issues between F5 and PA

RRAPP — Thu, 18 Feb 2016 13:33:32 GMT

After migrating from an ASA to PA3020, users reported that web pages were not fully loading. The issue was seen on the ASA but rarely. The PA3020 has been showing this issue more often than not resulting in a work around being done on the webpage. The trouble appears to be tied to how the F5 and Palo communicate. With caching enabled, the problem becomes intermittent. With caching disabled, the site fails regularly. F5 support has referenced a bug within the Palo ASIC: https://www.reddit.com/r/networking/comments/3eshjz/update_tcp_zerowindow_issues/

In working with Palo support, they have stated this is not related as we are running 6.0.10 and the article is relating this to 6.1.5.

To troubleshoot the issue without impacting production, a test environment was created and the issue can be replicated. Prior to inserting the Palo into the mix, the sites did not experience issues. Packet captures on the link show multiple out of sync packets when the page hangs. This is a concern as the only traffic being generated in the test environment is when we are testing which is going through the websites.

Has anyone seen this issue or know of anything similiar which can point me into the right direction to resolve?

Re: Web page issues between F5 and PA

pankaku — Thu, 18 Feb 2016 17:23:32 GMT

Issue seems to be asymmetric routing. Do pacps on firewall to confirm. Check the following documents:

https://live.paloaltonetworks.com/t5/Configuration-Articles/SYN-ACK-Issues-with-Asymmetric-Routing/ta-p/54090

https://live.paloaltonetworks.com/t5/Management-Articles/Packets-are-Dropped-Due-to-TCP-Reassembly/ta-p/57139

https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Set-the-Palo-Alto-Networks-Firewall-to-Allow-non-Syn/ta-p/62868

https://live.paloaltonetworks.com/t5/Featured-Articles/DotW-Issues-with-Asymmetric-Routing/ta-p/65456

Re: Web page issues between F5 and PA

pankaku — Thu, 18 Feb 2016 17:24:20 GMT

Check following document for pacps:

https://live.paloaltonetworks.com/t5/Learning-Articles/How-to-Run-a-Packet-Capture/ta-p/62390

https://live.paloaltonetworks.com/t5/Documentation-Articles/Packet-Based-Troubleshooting-Configuring-Packet-Captures-and/ta-p/54947

Re: Web page issues between F5 and PA

RRAPP — Thu, 18 Feb 2016 19:56:46 GMT

Thanks for the links as this helped confirm no packets are being dropped due to asymmetric routing. The FW was placed into bypass mode globally after trying by zone and the problem can be reproduced. The site does not fail all the time which is what makes this rather odd. The page also seems to stop loading at the same point which our application team is looking into.

Can anyone think of something else which can be check? The F5 and site coding has not been ruled out as of yet either. Would be nice to eliminate something in this mix.

Fun stuff.

Re: Web page issues between F5 and PA

santonic — Fri, 19 Feb 2016 08:07:04 GMT

Set a packet capture filter for this traffic on all stages. Compare packets received on ingress interface of PA, firewalled and transmited on the egress interface. This way you can make sure if PA is dropping any packets. In drop stage you shouldn't see any packets relevant for the observed TCP session.

At the same time do packet capture on web server as well. Check if all packets arrive to web server. I don't know about F5 packet capture capabilities. But I would do similar capture as on PA on F5 as well if possible. Do them siultaneuosly to compare them for same session.

This should give you the answer what is happening (or at least where are packets dissapearing).

Re: Web page issues between F5 and PA

RRAPP — Fri, 26 Feb 2016 13:55:41 GMT

Got pulled away to build out a third environment and hopefully will get back on it soon.