Palo Alto upgrade to 8.0.7 broken?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Palo Alto upgrade to 8.0.7 broken?

L3 Networker

Hi,

 

PA-3020. Had a customer upgrading from 8.0.4 to 8.0.7 and, after upgrading, autocommit failed with the message: 

"Failed to find address US";  "Unknown address US"; "Failed to parse security policy", where US is a default region object used in the security policy. There are no customer objects created in the Objects -> Regions section.

Checking here first as maybe there's already known problem with this? I see there's a post from a guy in the reddit about such a result as well, so that shouldn't be limited to a single device only.

Upgrade to 8.0.6-h3 with the same config was successful.

12 REPLIES 12

Cyber Elite
Cyber Elite

this is a configuration issue

 

you can go to the objects tab and go into the regions, then create a new object called 'US' and set it to the US region

 

this is possibly a relic that may not have been checked for, but for a definitive answer I'd advise opening a ticket with TAC

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

Objects -> Regions allow creating custom regions based on the IPs and/or location, but there are already predefined locations and they do not require any additional configuration. Or do they suddenly? 

https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Configure-a-Security-Policy-to-Us...

https://live.paloaltonetworks.com/t5/Community-Blog/Geolocation-and-geoblocking/ba-p/188812

 

@nikoo,

Potentially 'configuration issue' wasn't the best choice of words, but in reality it is. Its simply an error reading the built in region, which can be caused by a few different things; I've ran into it a few times in the past and other users have reported it as well. If you simply create the Region as @reaper stated it should fix the issue. 

You could also try running 'debug device-server reset id-manager type vsys-region' which resets the ids, that may get it enough to go through correctly.  

L4 Transporter

Apparently this is a known issue. If you ask me... due to the fact that the firewall does not come back up in a useable state this release (8.0.7) should be pulled and replaced with a fixed version. Since these are built in pre-defined objects it is *not* a config issue, something in 8.0.7 is broken and rolling back to any previous release works fine.

 

Here is the advice I recieved after submitting a very detailed report on this:

 

This issue seems due to conflict in name of the region [here it is US] in between vsys region and global region. Even though name might be removed from a vsys
region idmgr still keeps the info unless resetting the id-manager using the following command.

==============================================
from operational mode: <debug device-server reset id-manager type vsys-region>
from config mode <commit force>
==============================================

 

This worked for me on a 220 however on a 5050 the first command would not run because "The dataplane is not ready" even though I wated about 30 minutes. I ended up rolling it back and will try something different later to get to 8.0.7

 

@hshawn,

I just want to be very clear here, because I think you may be representing this as a bigger issue than it is, this issue is not specific to any one version of the OS. This is an issue that has been lingering for quite a while and is not something new with the 8.0 code. I myself have experienced this across multiple different platforms across 7.0, 7.1, and 8.0.

This is a known issue in the fact that it goes back a rather long way and can be caused by a number of configuration changes that get made. This is not a 8.0.7 specific issue. 

 

I'm not trying to be an ass about pointing this out, which this may come across as. However I don't want someone to stumble across this post and think that 8.0.7 is going to cause them issues when 8.0.6-h3 and 8.0.7 address some rather serious security concerns. 

@BPry

 

Interesting... it sure seems linked to this particular version of PANOS since it gets triggered after an update to 8.0.7 and then goes away if you roll back and then triggered again if you try 8.0.7 again and then goes away after rollback again 🙂 Do they not know what causes this? I have seen this on 3 seperate devices now when making the move from a 7.1 release to 8.0.7 and from an 8.0.6 and 8.0.6h3 release to 8.0.7

 

Anyway, for those encountering this try the suggested CLI commands yo umay have some luck

@hshawn,

More than likely it's something that simply isn't converting well when you upgrade to 8.0.7, generally related to specific configurations. It's important to note here that when you 'rollback' your switching to a different partition, so if sysroot0 is your 8.0.7 install and you rollback to 8.0.6 you are actually switching to sysroot1 and rewriting the system files. So when you keep trying to go back and forth between releases you won't really be fixing anything.

I can almost guarantee that if someone looked at the config that was trying to commit with 8.0.7 you'll find that it didn't move over correctly. This is recorded throughout many different software releases. 

As I already pointed out prior, and you reinterated, the command to fix this is 'debug device-server reset id-manager type vsys-region'. As a side point you should never jump into recommending someone use the 'commit force' command, if that was recommended to you by TAC then they were wrong in doing so. If the above debug command functioned properly then there would be no reason to force the commit, doing so can cause more issues than it can solve. 




@BPry

Thanks for the elaboration! We hav eseen config issues after an upgrade before but this is the only time we have seen it cause the device to fail to auto commit after coming back up. Also, good to know about the commit force they recommended, something to note... as soon as both of those commands were completed the interfaces came up and everything functioned normally on a lab 220. would you recommend issuing a reboot command in place of the commit force? Assuming it would reboot and then do another auto commit which should work after running the first command.

 

TIA

@hshawn,

I would simply recommend running a normal 'commit' once the debug command has been run and it should commit normally, once the debug command has been issued your validation error should clear up; the reason I don't recommend a commit force is because you lose any validation of the configuration. If you by chance modified something that would truly break the configuration a commit force would allow it to commit, where a normal commit will tell you the config is invalid and prevent the configuartion. 

The only time that a commit force should be used is if the validation is failing and you know for a fact that the config is valid. Even then I recommend exporting the canidate-config and going over it with a fine-tooth comb to verify that it isn't going to cause any issues before you actually issue the commit force command. 

We are still seeing this issue, so far we have been unable to get past this and upgrade to anything >= 8.0.7

 

PA220 - cli "fix" commands worked to get past geo object errors and device came up and works with the config unchanged, even upgraded from 8.0.7 to 8.0.8 no issues 

 

PA5050 (HA pair) - cli "fix" commands do not work (Dataplane not ready message every time), commit do not work (auto commit just keeps trying and failing over and over again and everything else queues behind it), deleting the policy from cli and doing a commit works but then we are unable to re-add the policy with the geo objects because it fails to commit with the same error in OP

 

 

 

 

L0 Member

@hshawn Thanks!

 

This solved the country code errors while upgrading to PANOS 8.0.8

==============================================
from operational mode: <debug device-server reset id-manager type vsys-region>
from config mode <commit force>
==============================================

@jeames

 

good! I'm glad you got past that error 🙂

 

For anyone wondering about us here, since these did not work on our 5050 HA pair devices I was able to do the following:

 

1. delete the security policies using the regional objects

2. upgrade

3. re-create policies

 

Please note step 1 has to be done before step 2. doing the upgrade and *then* deleting the policy will results in an upgrade and the auto commit will work but re-adding the policies will fail with the same error as before.

 

thanks all!

  • 5574 Views
  • 12 replies
  • 1 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!