Seeking palo troubleshooting advice

[MikeB]

Im involved in a project to migrate away from old asa firewalls to a palo solution.

The process has gone well but myself and peers are stumped with an odd issue and looking for troubleshooting advice.

We have a number of https hosts in a dmz, nat'ed to be available to the public internet. systems from all over the world can access these https hosts fine. That is.... except for a small group of individuals all using a particular local ISP. The trouble is, this issue did not exist with the previous ASA firewalls.

The security policy is simple, allow tcp 443 in. People with this issue will intermittently have difficultly fully loading an https site behind our palos. Pages will partially load or time out. Chrome debug logs from the end users perspective show timeouts on connection attempts.

If a user with the issue retries enough they will eventually establish a session and have no troubles. The https hosts in our DMZ show connection attempts from users on this ISP and from their perspective traffic seems to just stop.

No drops or blocks appear in threat or security logs on the palo and we dont see this behavior with any other people from other ISPs.

The ISP has insisted they are not mangling traffic and from what Ive seen I agree but the problem persists. Ordinarily Id write this off as an ISP doing something odd but the problem didnt exist when our prior ASA firewalls were in place.

The issue feels like a content inspection or ssl decryption issue but we've confirmed thats not in play. We've also confirmed any outside IDS systems are not mangling or dropping traffic. We dont see any asyc routing in play so it does not appear the palo is loosing state and dropping traffic. Everything from what we see appears the same as it was but still we continue to have issues with people using this one particular ISP

Myself and peers are looking for further troubleshooting insights or advice.
Any advice would be greatly appreciated.
-Thanks

[MikeB]

LOL thats pretty good. 

Thanks for the ACK advice! Sounds plausible. Ill check it out.

[jesseivens]

Use TAC, that is what they are there for.  Take advantage of the service you paid for 🙂  

[MikeB]

I peeked at our palos at each datacenter one has the setting turn on, the other has it turned off. 

eg:

show running tcp state
Allow Challenge Ack : yes

show running tcp state
Allow Challenge Ack : no

Back to the drawing board unfortunately... that challenge ack article really seemed promising