What is "FIPS failure. Power-On Integrity Self Test Failure (FS)"?

[emr_1]

My customers are facing critical issue when he upgrades firmware.

One customer is using VM-100, when he upgrades from 8.1.0 to 8.1.10 and reboot the device, he sees this issue.

Another customer is using PA-500, when he upgrades from 8.1.0 to 8.1.9-h4 and reboot the device, he sees this issue.

Both customers upgrades from 7.1.x and 8.0.x, and all steps until upgrading to 8.1.0 are fine.

Even the reason points 'FIPS', they are not using FIPS mode.

Anyone knows the cause?

Note: We can not proceed factory reset after we see the issue. It fails.

Regards,

Emr

[BPry · Accepted solution]

@emr_1,

I would open a TAC case so that they can log the issue and get the full upgrade path to see if they could potentially recreate the issue. If this was just on the PA-500 I would be leaning more towards hardware failure, but with it being on a VM-100 alongside a PA-500 it's possible that a bug is present in the upgrade path that was followed. 

[Egghead_Systems]

HI @emr_1, we are also experiencing a similar error. When our customer tried to upgrade from 8.0.11-h1 to 8.1.9-h4; their PA3020 went to Maintenance Mode after installing and rebooting .

The Maintenance Mode simply stated that there is a "FIPS failure".

The upgrade steps that we followed are:

a) Download 8.1.0 (base) , without installing

b) Download and Install 8.1.9-h4 

After we did step b above the PA3020 rebooted and went straight to maintenance mode with error "FIPS failure"

Luckily, we were able to revert back again to 8.0.11-h1. But , we still need to upgrade to 8.1.x, becuase 8.0.x is already EOL.

We have already contacted palo alto TAC and are now waiting for their reply.

While we are waiting for pan tac reply, would you mind sharing what happened with your situation? How did you guys resolve the FIPS error?

any feedback would be great, thanks

glenn

egghead systems

[emr_1]

Hi @Egghead_Systems ,

First of all, the reason you and I went to maintence mode is because of new feature that installed from 8.1.1, called "

Software Integrity Check".

You can find this under release note.

https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-release-notes/pan-os-8-1-release-information/fea...

As TAC told me, PAN-OS detected some files were broken, such it stoped normal start up operation and went to maintence mode.

After you see entering maintence mode, you can find log from fips.log on the menu and see which specific file was failed.

The log below is my sample:

===

10/14/19 15:31:09 fips    ERROR: failed integrity check on /etc/pan-manifest/mgmt-panos(//var/appweb/sslvpndocs/global-protect/getsoftwarepage.esp: FAILED)

10/14/19 15:31:09 fips    ERROR: FIPS-CC integrity on fs:Management plane failed verification on 1 files.

10/14/19 15:31:09 fips    ERROR: * * * * *  FIPS Self-Tests failed * * * * *

10/14/19 15:31:41 fips    ERROR: * * * * *  FIPS Self-Tests (**panic**) trying os command * * * * *

===

From above situation, we can take two ways:

1) try to proceed factory reset and see broken files are replaced by original files (don't forget to take your config backup before you proceed)

2)open the ticket, and request RMA

On my case, we did RMA... I could not escape from maintence mode and found no way.

[Egghead_Systems]

@emr_1 Thanks for your reply. When you got your RMA, what was the PANOS that came with it? was it 8.1.0 already?

Also, was your palo alto a pa500?

thanks

[emr_1]

@Egghead_Systems  Yes, replacement was 8.1.0. And also, my cases were PA-500 and VM-100. On both cases, the cause and result of issue were same, but broken file was different. Hope it helps you.

[jade.jisman]

I'm experiencing the same issue. Can someone hlep me on this?

What I did was, 

I upgraded the 8.0.x

download base 9.0.0

download and install 9.0.9-h1

I just factory reset the firewall since I'm able to gain login access

and now  we have this issue.

Please we need urgent assisntance

ATTENTION: A critical error has been detected preventing proper boot
up of the device. Please contact Palo Alto Networks to resolve this
issue.

866-898-9087 or support@paloaltonetworks.com

Entry Reason: FIPS failure.
See 'Entry Reason' for more information.

< Continue

[ahmadzubair654]

Hello,

Were you able to figure out what was the issue? How did it resolve?

[Salathiwe]

Hey there

Did you manage to get the issue resolved?