TCP issues when moving an application through a Palo Alto FW

[CarloTaddei]

Hi,

Following scenario:

we have a 2-level Firewall Filtering / Security Setup active in our infrastructure, with a Cisco ASA currently acting as the Internet Firewall (updated to the latest Cisco ASA OS version) and an internal Firewall (Checkpoint appliance, also updated to GAIA OS R80.20). Among others (like Web Servers, DNS, Email, and so on), an Internal Webserver application is hosted after the internal Firewall and made accessible from the Public Internet (with IP Src address filtering - without IPSec) via the 2 above mentioned Firewalls Levels. 

We are currently in the Process of migrating the Cisco ASA Firewall to a Palo Alto Firewall (PAN OS vers. 8.1.0); the physical appliance that we chose as our Internet Firewall is extremely oversized for our needs - so possible resources limits being reached for what is about to be mentioned should really not be the case here (overall FW usage is less than 1% ...) ...

We have a (strange and unexpected) issue with an application (already active in our network, and working without any issues whilst passing through the Cisco ASA) that we systematically experience when moving the application through the new PALO ALTO FW:

CLIENT <--------->  INTERNET <------------> CISCO ASA <-----------> Checkpoint <-------------> Web Server Hosting the Application

(Traffic Flow working from Client to Server without any issues)

CLIENT <--------->  INTERNET <------------> PALO ALTO <-----------> Checkpoint <-------------> Web Server Hosting the Application

Sessions are allowed and created from Client to Server (via the PALO ALTO), and properly passed downwards the chain; anyhow the Checkpoint FW starts having issues handling properly the TCP sessions opened through the Palo Alto FW: the Checkpoints seems to be dropping a fraction of the return Traffic (from Web Server to Client) for a still unknown reason and somehow systematically (not all sessions are dropped).

At first we tought / assumed a Checkpoint related Firewall Issue -> however what really puzzles us is that as soon as we start routing traffic via the Cisco ASA, connectivity is immediately fully restored and no drops whatsoever are seen on the checkpoint logs anymore ... so this somehow leads us to the idea of a different TCP sessions handling behaviour between the Cisco ASA and the Palo Alto FW.

On the Palo Alto FW we are not implementing anything special yet - just allowing tcp / 443 inbound.

Possible Asymmetric routing issues should also not be the root cause for this issue.

Any insight / suggestions on possible TCP related session handling issues on the PALO ALTO would be highly appreciated.

Thanks in advance.

[reaper]

Have you tried setting up packet-diag filters and tracing the global counters to see if maybe any attributes are being stripped off? Is there any indication why the CP is dropping some packets? Is there fragmentation? (MTU mismatch)

[santonic]

It's a very strange issue.

I would try capturing traffic at multiple points and comparing them:

- ingress interface of internet FW, 

- egress interface of internet FW, 

- ingress interface of internal FW, 

- egress interface of internal FW.

Maybe on server as well.

[TSilverline]

Check Point should be able to tell you why it's dropping the packets.  I would start there and figure out what it doesn't like and then determine how that is happening to the packets.