This is not an issue, but a general document about an issue that we experience with a customer last weekend. The issue is not well documented by Palo TAC and it took us the help of another customer who experienced the same issue with the same application vendor. One of our Electronic access systems stopped working after changing the perimeter firewalls to PA-5260 over the last weekend. The EAC has controllers (used for building accesses) that need to contact a server in the cloud. Simple configuration on the fw perspective allowing policy, NAT for the internal to external communication. We experienced issue with connecting to server after placing the PA-5260 firewall in the environment and started seeing Server send RST-ACKs in the packet-captures. the session teardown reason being - tcp-rst-from-server After taking capture on the server, firewalls and the controllers, we saw a strange behavior on the firewall. Tracing a single TCP stream, we see the controller sends 37 packets to the server, but the firewall ingress interface only receives 36 packets and sends 36 packets to the server. The client keeps send a re transmission packet, since it didn't get any ACK for the PUSH-ACK packet that did not reach the firewall. And after waiting for 14 seconds the server sends a reset. Deeper investigation into the issue reveals that the packet that does not make it to the firewall is the packet which has a VSS-Management trailer added at the back by the Controllers. The root cause is with newer hardware models that use the FPGA FE100 hardware chip, which causes the firewall drops certain segments containing a VSS-Management trailer. This is due to the firewall performing an FCS on ingress, but the added VSS-Management trailer breaks the checksum and the segment doesn't make it to the destination. Taking global counters on the firewalls yielded below output: > show counter global filter delta yes | match L4 [Kflow_fpga_rcv_igr_L4CHKSUMERR 46 5 info flow offload FPGA IGR Exception: L4CHKSUMERR appid_ident_by_dport_first 1386 177 info appid pktproc Application identified by L4 dport first appid_ident_by_dport 38 4 info appid pktproc Application identified by L4 dport We see firewall dropping the packets due to checksum failure and hence not making it to the dataplane. Disable Layer 4 Checksums Perform the below on both firewalls using HA to minimise any impact. i.e passive first. 1. On the Firewall, disable layer4 checksum using below command: > set system setting layer4-checksum disable 2. Reboot the device. 3. After box comes up after reboot, confirm setting in sdb: > show system state | match fe100 Result: You should be getting l4_chk_sum': 0 as below: cfg.hw.fe100: { 'cfg_mode': 10, 'l4_chk_sum': 0, 'usecase': 1, 'v4_v6_choice': 2, Since L4 checksum will no longer be performed on the firewalls, TCP retransmissions due to invalid checksum would still occur because of the server/client checksum validation. Related documents: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLpICAW https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-release-notes/pan-os-8-1-addressed-issues/pan-os-8-1-5-addressed-issues
... View more