PAN-OS-11.1.2-h3 - No incomming traffic after upgrade

[Harikrishnan.P]

Hi,

We recently upgraded our Palo Alto 1410 Firewall to PAN-OS-11.1.2-h3 from PAN-OS-11.0.4-h1.

After Upgrade there was no incoming traffic from external networks. There were no hits or logs showing incoming traffic.

Internet Outbound traffic was going through normally.

IPSEC VPN tunnels were working normally.

Support team checked and wanted us to downgrade to the previous version.

Is this a bug in PAN-OS 11.1 ?

Has anyone ever faced this issue after PAN-OS upgrades ?

Should we install the base image for 11.1 before we upgrade to 11.1.2-h3?

Any ideas and suggestiions are welcome.

Thanks

Hari

[reaper]

Can't say i've encountered this issue before due to a bug, but have seen similar things happen due to ARP issues. have you checked upstream MAC and ARP tables, are arp requests for the public IP of the firewall being replied to when inbound packets arrive, are tables updated accordingly?

you could set up packetcapture and follow global counters to see what is happening on the firewall side, also packewt capture the upstream device and see what's going on

in regards to upgrading: you don't need to install the base image, it just needs to be downloaded for you to be able to install maintenance packages

[Harikrishnan.P]

Thanks for the update. We hace asked for support from Palo Alto as well. Once i successfully upgrade our 11.0 image to 11.1 image, will keep you updated on the procedure.

[zahmed01]

Hello,

Please have the support team open a ticket with engineering to have them take a look into the issue and obtain the next steps from them. Also as mentioned earlier, base image only needs to be downloaded during the upgrade process. 

Thanks,

[Harikrishnan.P]

Hi,

No incoming traffic after trying to upgrade 3 different versions of PAN-OS with PA1410 firewall.
On all the 3 occassions we had to revert back to the version which was running on the firewall.
There were no log hits on the "Monitor" showing incoming traffic.
We did see some icmp traffic but not http / https traffic.

From PANOS 11.0.2-h2 to PANOS 11.1.0 - no incomming traffic after upgrade
From PANOS 11.0.4-h1 to PANOS 11.1.2-h3 - no incomming traffic after upgrade
From PANOS 11.0.4-h1 to PANOS 11.0.5 - no incomming traffic after upgrade

We were able to upgrade a version inbetween from 11.0.2 to 11.04 without any issues.

Palo Alto support is still working on the issue.

Any ideas ?

Thanks
Hari

[leonnl]

I have the exact same issue on a PA-1410 HA pair. ARP is not updating. If you are in a HA pair, clear the arp manually and see that the arp table is no longer populated after the upgrade.

I've upgraded from 11.0.4-h1 to 11.0.5 (no arp), same for 11.0.4-h1 to 11.1.4-h1 (no arp). Something is seriously wrong, I guess it's PA-1410 related.

[Z.Boussaid]

this issue has been fixed in 11.1.2-h9 release. 

https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-release-notes/pan-os-11-1-2-known-and-addressed... 

I'm running 11.1.2-H3, traffic is available with no issues.

[JamieEnglish]

I'm running 11.1.2-h3, and I see no logs in my Panorama instance. We did try to upgrade to 11.1.2-h9, and we got an error:

• Failed to install 11.1.2-h9 with the following errors.
• SW version is 11.1.2-h9
• Nothing pending to cancel
• Error: Traceback (most recent call last):
• File "/opt/panrepo/releases/11.1.2-h9/validate", line 340, in <module>
• inds = check_for_old_indices()
• File "/opt/panrepo/releases/11.1.2-h9/validate", line 146, in check_for_old_indices
• check_dir_for_es_ind(in_dir)
• File "/opt/panrepo/releases/11.1.2-h9/validate", line 125, in check_dir_for_es_ind
• (name, ver, cdate) = get_es_ver_cdate_of_idx(fullpth)
• File "/opt/panrepo/releases/11.1.2-h9/validate", line 116, in get_es_ver_cdate_of_idx
• return (name, ver, cdate)
• UnboundLocalError: local variable 'cdate' referenced before assignment
•  
• Failed to install version 11.1.2-h9 type cms

We opened a case with TAC, and they said we need to wait until 11.1.5 comes out. 

[Z.Boussaid]

The error message you’re encountering during the installation of PAN-OS 11.1.2-h9 on your Palo Alto firewall points to a problem in the validation script, specifically an UnboundLocalError caused by a variable cdate being referenced before it's assigned. try this:

1. Verify Compatibility
   Ensure that PAN-OS 11.1.2-h9 is compatible with your firewall model. Check the Palo Alto Networks documentation for any compatibility notes or prerequisites.
2. Clear Previous Installation Attempts
   Sometimes, remnants from previous installation attempts can cause issues. Ensure that there are no partial installations or residual files. You might need to clear out old files or directories related to previous installations.

3. Cancel Pending Jobs: Even though you mentioned nothing pending, double-check that there are no pending jobs or incomplete installations.

   Check and Clean Installation Directory: If applicable, look for and clean up old installation directories or files in /opt/panrepo/releases/.

4. Update or Re-download the Software Image
   The image you downloaded might be corrupted. Try downloading the PAN-OS 11.1.2-h9 image again from the Palo Alto Networks support site.
5. File System Check
   Ensure there is enough free space on the firewall’s file system. Lack of space can sometimes lead to incomplete installations and script errors.

i'm running the 11.1.2-H9 on my new 1420 in HA with traffic flowing. 

[JamieEnglish]

Thanks!

On Step 2 and 3, do you have a list of commands to run for those?

[Z.Boussaid]

Step 2:

1. To cancel pending jobs or incomplete installations on a Palo Alto firewall, you can use the following CLI commands:
   1. Check for Pending Jobs: show job all   - This will display all jobs, including their status. Look for jobs with statuses like pending, running, or stalled.
   2. Cancel a Specific Job: If you find any jobs that are stuck or you want to cancel, note the job ID from the output of the previous command. Then use the following command to cancel the specific job: request job cancel <job-id> 
   3. Cancel all jobs: request job cancel all. Be cautious with this command as it will cancel all ongoing or pending jobs.
   4. Check System and Installation Status: To ensure the system is in a stable state and to check if any installation is in progress, you can use: show system info .This command provides an overview of the system status, including current software version and installation status.

Step 3:

1. List Directory Contents:

   ls /opt/panrepo/releases/

2. Check Disk Space:

   show system disk-space

3. Remove Old Files (Be Cautious):

   rm -rf /opt/panrepo/releases/old-version-directory

4. Verify Current Software Version:

   show system info

5. Check for Ongoing Jobs:

   show job all

Always be cautious when using rm -rf as it can permanently delete files and directories. Ensure you only remove files or directories that are no longer needed and not required for current operations. If in doubt, consult Palo Alto Networks support or documentation. good luck

1.  

[JamieEnglish]

I appreciate it. Looks like Step 2 is squared away. However, when running any ls commands, I get this:
admin@PAN-01> ls
Unknown command: ls
admin@PAN-01>

[Z.Boussaid]

i think the ls command is accessed on the Shell level on PA, which is disabled and only TAC can have access to it.. this to prevent the rm -rf commands from being executed. i think if you need the list you might need to have TAC check on it for you. the PAs can be rooted, but it will void your warranty and support which I'm sure you don't want. so if the command did not work it is because of the root access limitation. 

[SCALTEL]

Hi,

i am also experiencing a similar issue:
Our panorama is on 11.0.5. When we want to upgrade, we are receiving the following error message:

Failed to install 11.1.4-h1 with the following errors.
SW version is 11.1.4-h1
Nothing pending to cancel
Error: Traceback (most recent call last):
File "/opt/panrepo/releases/11.1.4-h1/validate", line 359, in <module>
(min([dts['min'] for dts in log_type_intv_dir.values() if dts['min']]).strftime('%Y-%m-%d'),
ValueError: min() arg is an empty sequence
Failed to install version 11.1.4-h1 type cms

Any ideas? I also opened a TAC case. Unfortunately they did not found a solution yet.

[Z.Boussaid]

i think you need to download 11.1.0 before you jump from 11.0.5 to 11.1.4-h1. do you have 11.1.0 downloaded in Panorama.