02-14-2022 08:43 AM
What seems to be out of the blue, with no configuration changes on our firewall(s), we began experiencing random periods of "network outages" on our main data center firewall. The symptoms are as follows:
Our pingdom test to our OWA website shows as down (i.e. the web page hosted behind the firewall cannot be reached from the internet)
Users connected via GlobalProtect experience either horrible performance or outright disconnects.
GP connected users running ping tests to internal resources show 10-60+% packet loss. (it varies widely)
Users connected in the data center running ping tests to ISP gateway show similar packet loss patterns and experience horrible performance / no connection to the internet.
This condition lasts 1-20+ minutes and is obviously causing major headaches.
I've opened a TAC case, but as seems to be the case as of late, the wait times to talk to a human are long.
We have an HA pair in an active/passive setup at the data center and I've failed over with no apparent change in this condition. I've also removed the in-line switch (that takes the one physical connection from our ISP and splits it to the HA pair) from the equation with no change in the condition.
I can't MAKE it happen, but it has been happening regularly enough to be a real thorn in our side.
I haven't seen anything in the system log that I recognize as being relevant to this behavior.
I upgraded the pair to PAN-OS 10.0.8-h8 in the hopes I'd tripped an obscure bug and the update would fix it. Sadly it did not.
If anyone as ideas, I am wide open to them.
