topic Re: Very weird PanOS upgrade issue in General Topics

Very weird PanOS upgrade issue

raji_toor — Wed, 22 Sep 2021 15:00:06 GMT

We have a pair of active passive firewalls on PanOS 9.0.11. We have attempted to upgrade to 9.1.10 4 times out of which 2 times was with tech support. Upgrade seems to cause issue with some linux servers which are critical to us. As an example we have system running Nagios which starts alerting immediately for number of devices. And when we try to ssh from a system we get the user prompt and most of the time we don't see the password prompt and only 1 or 2 times may reach password prompt but authentication fails.

It causes issue between 2 linux servers in the same vlan. The only reason i can think of this night be caused by is that because these are linux boxes authenticate with a DC which is in a different zone.

Also SSH alone is not the issue, cloud monitoring systems report couple of linux system running some web services down. And an Oracle database fails.

Re: Very weird PanOS upgrade issue

kiwi — Thu, 23 Sep 2021 14:16:29 GMT

Hi @raji_toor ,

It would be more helpful if you could share some info on how the traffic is being identified by the FW.

Are you seeing any drops or strange global_counters acting up ?

Cheers,

-Kiwi.

Re: Very weird PanOS upgrade issue

BPry — Fri, 24 Sep 2021 01:43:30 GMT

@raji_toor,

As @kiwi mentioned the information being provided isn't enough to actually help you troubleshoot these issues. When you've attempted these upgrades have you had your interzone-default policy setup to log denied traffic and ensured that you aren't dropping anything due to the upgrade causing the app-id being identified to change?

When you ran through this with support what did you guys verify post change and what did they identify as possible issues? They should have grabbed a technical support file before letting you revert the upgrade so they at least had a copy of the logs to look through to aid in investigation. I'd also recommend in the next attempt that you ensure you have syslog forwarding setup for all traffic and threat logs so they're being forwarding to something off the device so you can review those as well when stuff isn't actively down.

Re: Very weird PanOS upgrade issue

raji_toor — Thu, 18 Nov 2021 05:04:32 GMT

After 3 months it seems issue has been identified and using commands in this article resolved the issue with passive firewall. Active will be upgraded and reset in the next outage window.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000boS5CAI