topic Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade in Next-Generation Firewall Discussions

PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Harikrishnan.P — Wed, 17 Jul 2024 09:49:56 GMT

Hi,

We recently upgraded our Palo Alto 1410 Firewall to PAN-OS-11.1.2-h3 from PAN-OS-11.0.4-h1.

After Upgrade there was no incoming traffic from external networks. There were no hits or logs showing incoming traffic.

Internet Outbound traffic was going through normally.

IPSEC VPN tunnels were working normally.

Support team checked and wanted us to downgrade to the previous version.

Is this a bug in PAN-OS 11.1 ?

Has anyone ever faced this issue after PAN-OS upgrades ?

Should we install the base image for 11.1 before we upgrade to 11.1.2-h3?

Any ideas and suggestiions are welcome.

Thanks

Hari

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

reaper — Thu, 18 Jul 2024 08:42:10 GMT

Can't say i've encountered this issue before due to a bug, but have seen similar things happen due to ARP issues. have you checked upstream MAC and ARP tables, are arp requests for the public IP of the firewall being replied to when inbound packets arrive, are tables updated accordingly?

you could set up packetcapture and follow global counters to see what is happening on the firewall side, also packewt capture the upstream device and see what's going on

in regards to upgrading: you don't need to install the base image, it just needs to be downloaded for you to be able to install maintenance packages

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Harikrishnan.P — Thu, 18 Jul 2024 09:40:15 GMT

Thanks for the update. We hace asked for support from Palo Alto as well. Once i successfully upgrade our 11.0 image to 11.1 image, will keep you updated on the procedure.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

zahmed01 — Fri, 19 Jul 2024 16:06:47 GMT

Hello,

Please have the support team open a ticket with engineering to have them take a look into the issue and obtain the next steps from them. Also as mentioned earlier, base image only needs to be downloaded during the upgrade process.

Thanks,

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Harikrishnan.P — Mon, 29 Jul 2024 05:27:46 GMT

Hi,

No incoming traffic after trying to upgrade 3 different versions of PAN-OS with PA1410 firewall.
On all the 3 occassions we had to revert back to the version which was running on the firewall.
There were no log hits on the "Monitor" showing incoming traffic.
We did see some icmp traffic but not http / https traffic.

From PANOS 11.0.2-h2 to PANOS 11.1.0 - no incomming traffic after upgrade
From PANOS 11.0.4-h1 to PANOS 11.1.2-h3 - no incomming traffic after upgrade
From PANOS 11.0.4-h1 to PANOS 11.0.5 - no incomming traffic after upgrade

We were able to upgrade a version inbetween from 11.0.2 to 11.04 without any issues.

Palo Alto support is still working on the issue.

Any ideas ?

Thanks
Hari

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

leonnl — Fri, 23 Aug 2024 18:44:04 GMT

I have the exact same issue on a PA-1410 HA pair. ARP is not updating. If you are in a HA pair, clear the arp manually and see that the arp table is no longer populated after the upgrade.

I've upgraded from 11.0.4-h1 to 11.0.5 (no arp), same for 11.0.4-h1 to 11.1.4-h1 (no arp). Something is seriously wrong, I guess it's PA-1410 related.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Z.Boussaid — Wed, 28 Aug 2024 17:37:31 GMT

this issue has been fixed in 11.1.2-h9 release.

https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-release-notes/pan-os-11-1-2-known-and-addressed-issues/pan-os-11-1-2-h9-addressed-issues

I'm running 11.1.2-H3, traffic is available with no issues.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

JamieEnglish — Wed, 04 Sep 2024 15:33:35 GMT

I'm running 11.1.2-h3, and I see no logs in my Panorama instance. We did try to upgrade to 11.1.2-h9, and we got an error:

Failed to install 11.1.2-h9 with the following errors.
SW version is 11.1.2-h9
Nothing pending to cancel
Error: Traceback (most recent call last):
File "/opt/panrepo/releases/11.1.2-h9/validate", line 340, in <module>
inds = check_for_old_indices()
File "/opt/panrepo/releases/11.1.2-h9/validate", line 146, in check_for_old_indices
check_dir_for_es_ind(in_dir)
File "/opt/panrepo/releases/11.1.2-h9/validate", line 125, in check_dir_for_es_ind
(name, ver, cdate) = get_es_ver_cdate_of_idx(fullpth)
File "/opt/panrepo/releases/11.1.2-h9/validate", line 116, in get_es_ver_cdate_of_idx
return (name, ver, cdate)
UnboundLocalError: local variable 'cdate' referenced before assignment
Failed to install version 11.1.2-h9 type cms

We opened a case with TAC, and they said we need to wait until 11.1.5 comes out.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Z.Boussaid — Wed, 04 Sep 2024 16:03:49 GMT

The error message you’re encountering during the installation of PAN-OS 11.1.2-h9 on your Palo Alto firewall points to a problem in the validation script, specifically an UnboundLocalError caused by a variable cdate being referenced before it's assigned. try this:

Verify Compatibility
Ensure that PAN-OS 11.1.2-h9 is compatible with your firewall model. Check the Palo Alto Networks documentation for any compatibility notes or prerequisites.
Clear Previous Installation Attempts
Sometimes, remnants from previous installation attempts can cause issues. Ensure that there are no partial installations or residual files. You might need to clear out old files or directories related to previous installations.
Cancel Pending Jobs: Even though you mentioned nothing pending, double-check that there are no pending jobs or incomplete installations.

Check and Clean Installation Directory: If applicable, look for and clean up old installation directories or files in /opt/panrepo/releases/.
Update or Re-download the Software Image
The image you downloaded might be corrupted. Try downloading the PAN-OS 11.1.2-h9 image again from the Palo Alto Networks support site.
File System Check
Ensure there is enough free space on the firewall’s file system. Lack of space can sometimes lead to incomplete installations and script errors.

i'm running the 11.1.2-H9 on my new 1420 in HA with traffic flowing.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

JamieEnglish — Wed, 04 Sep 2024 16:18:15 GMT

Thanks!

On Step 2 and 3, do you have a list of commands to run for those?

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Z.Boussaid — Wed, 04 Sep 2024 16:46:38 GMT

Step 2:

To cancel pending jobs or incomplete installations on a Palo Alto firewall, you can use the following CLI commands:
1. Check for Pending Jobs: show job all - This will display all jobs, including their status. Look for jobs with statuses like pending, running, or stalled.
2. Cancel a Specific Job: If you find any jobs that are stuck or you want to cancel, note the job ID from the output of the previous command. Then use the following command to cancel the specific job: request job cancel <job-id>
3. Cancel all jobs: request job cancel all. Be cautious with this command as it will cancel all ongoing or pending jobs.
4. Check System and Installation Status: To ensure the system is in a stable state and to check if any installation is in progress, you can use: show system info .This command provides an overview of the system status, including current software version and installation status.

Step 3:

List Directory Contents:

ls /opt/panrepo/releases/
Check Disk Space:

show system disk-space
Remove Old Files (Be Cautious):

rm -rf /opt/panrepo/releases/old-version-directory
Verify Current Software Version:

show system info
Check for Ongoing Jobs:

show job all

Always be cautious when using rm -rf as it can permanently delete files and directories. Ensure you only remove files or directories that are no longer needed and not required for current operations. If in doubt, consult Palo Alto Networks support or documentation. good luck

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

JamieEnglish — Wed, 04 Sep 2024 17:09:53 GMT

I appreciate it. Looks like Step 2 is squared away. However, when running any ls commands, I get this:
admin@PAN-01> ls
Unknown command: ls
admin@PAN-01>

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Z.Boussaid — Wed, 04 Sep 2024 17:48:22 GMT

i think the ls command is accessed on the Shell level on PA, which is disabled and only TAC can have access to it.. this to prevent the rm -rf commands from being executed. i think if you need the list you might need to have TAC check on it for you. the PAs can be rooted, but it will void your warranty and support which I'm sure you don't want. so if the command did not work it is because of the root access limitation.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Marco_DiFrancesco — Mon, 23 Sep 2024 12:07:27 GMT

Hi,

i am also experiencing a similar issue:
Our panorama is on 11.0.5. When we want to upgrade, we are receiving the following error message:

Failed to install 11.1.4-h1 with the following errors.
SW version is 11.1.4-h1
Nothing pending to cancel
Error: Traceback (most recent call last):
File "/opt/panrepo/releases/11.1.4-h1/validate", line 359, in <module>
(min([dts['min'] for dts in log_type_intv_dir.values() if dts['min']]).strftime('%Y-%m-%d'),
ValueError: min() arg is an empty sequence
Failed to install version 11.1.4-h1 type cms

Any ideas? I also opened a TAC case. Unfortunately they did not found a solution yet.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Z.Boussaid — Mon, 23 Sep 2024 12:14:54 GMT

i think you need to download 11.1.0 before you jump from 11.0.5 to 11.1.4-h1. do you have 11.1.0 downloaded in Panorama.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Marco_DiFrancesco — Mon, 23 Sep 2024 12:22:33 GMT

Yes the base image is downloaded.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

ttsws — Tue, 01 Oct 2024 09:09:22 GMT

Same here.

Panorama upgrade fails in install after a few seconds.

11.0.3-h5 -> 11.1.4-h1 fails.

11.1.0 -> 11.1.4-h1 fails

11.1.0 -> 11.1.3-h6 fails

The error message are the same.

Still waiting for an analysis from our partner.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

Tobi — Wed, 02 Oct 2024 07:36:20 GMT

Any news on the TAC case?
We are experiencing the same problem.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

ttsws — Wed, 02 Oct 2024 11:13:37 GMT

ASC TAC says. they found similar issues. The solution was to delete all software images and redownload the required ones.

This did not help in our case.

Recommendation was to upgrade via CLI to get more error output, but in our case the output was the same which is displayed by the GUI.

I hope, they will escalate this to PAN TAC.

Re: PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

ttsws — Sat, 19 Oct 2024 19:08:18 GMT

Had a session with Palo TAC. root level access at the beginning of the week. Expected solution did not work. Next solution: reinitialize Elastic Search, losing all logs. Not an option. Palo engineer wanted to check with colleagues for other workarounds. Yesterday I got new suggestions, which I have not tested, yet: Upgrade to 11.0.5 or 11.06 and then to 11.1.4 or go directly to 11.1.5 which fixes PAN-258757.