Antivirus Dynamic Update fails PAN-OS 8.1.0 Cluster

[Chacko42]

Hi Community,

I have a PA-850 Cluster with PAN-OS 8.1.0 and a valid Threat license.

The active firewall is configured to download and install antivirus updates and sync them to his peer.

Unfortunately, the update failed lately, so we were 4 days behind the current versions.

After manually using "check now" the new updates were found without problems.

In the system log the update-lookup was logged during the scheduled time, but there were no updates found.

After looking in the ms.log during these time period I saw these entries:

'cfg.fail-conn-on-cert': NO_MATCHES
NO_MATCHES
NO_MATCHES
/tmp/.avinfo.xml.11208:1: parser error : Start tag expected, '<' not found
The service is unavailable.
^
2018-04-05 13:15:39.368 +0200 Error:  pan_file_to_xml(pan_xml_utils.c:550): error parsing file /tmp/.avinfo.xml.11208

Does anybody experienced the same behavior?

Manually installing the updates once doesn't solve the problem

Best Regards

Chacko

[BPry]

@Chacko42,

So just to be clear and ensure that I'm understanding this correctly; when you manually update everything finishes correctly, but even once it has been updated you continue to run into issues using the update scheduler? 

Did this happen after you upgraded to 8.1.0 or has this been constant through 8.0.* and 8.1.0? 

[Chacko42]

@BPry - it's a new system, we directly went to 8.1.0 because of the hit counters.

This morning there was an information by PaloAlto regarding a similar issue

https://live.paloaltonetworks.com/t5/Customer-Advisories/Content-Update-Advisory-Important-Informati...

Apps & Threats are on 8000-4618 and Antivirus is now manually on 2571-3067.

The firewall will check again at 13:15 - I will update the status after reviewing the logs.

[Chacko42]

Well, the manual update unfortunately didn't fix the problem.

Importing the data file manually didn't worked as well.

I opened up a case and we will see what's going on.

[RyanGates]

This exact issue is happening for our PA-820's setup for HA. However, it is also happening for our PA-220 not setup for HA. All three of these firewalls are running 8.0.8 though. Manual check and download did fix the issue for these three firewalls.

We do have on other firewall running 7.1.5 PA-200 that has no issues download and installing updates on the schedule. Seems to be related to the PANOS verison. Not sure if something changed in regards to Dynamic updates from 7.1 to 8.0 but something is wrong. I know the 3 digit to 4 digit issue they sent an email about but this was happening before that update for us and after the upgrade in PANOS to 8.0.

[RyanGates]

I checked all three firewalls and I did find a difference between the two different PANOS's. This was unchecked on 7.1.x and Checked on 8.0. Wonder if this is the issue with dyynamic updates.

[Chacko42]

@RyanGates: I thought the same because of the

'cfg.fail-conn-on-cert' 

 in the log - but the certificate chain is trusted and the root certificate for updates.paloaltonetworks.com is stored on both nodes.

[RyanGates]

From another thread seems like when you change the Schedule time, this fixes the issue. I am going to test that out tonight.

Also, unchecking that box did not fix the issue.

[Chacko42]

@RyanGates yeah, I tested that as well, but that didn't work.

When I set the primary firewall to download and install and sync-to-peer, everything is fine.

But as soon as the secondary firewall tries to look up to the updates on its own, the posted logs are occuring and the update fails, even if the GUI logs look good.

[Chacko42 · Accepted solution]

Ok, I have a solution.

Both cluster nodes were configured to download the dynamic updates on their own - we already configure the timers, so there is a little delay between the downloads.

Nevertheless, there was a donwload collision with some other scheduler.
The Tech Support analyzed the log files and told us to reschedule the updates - that worked indeed.

I guess it would be a good idea to create a clever log message if these things happen, so that administrators do not need to open tickets for that. Furthermore it's possible by configuration to schedule all of the dynamic updates to download and install at 0 minutes after each hour, so it's a little bit poor, that a NGFW cannot offer a decent queue/sync to deal with these issues...

Best Regards

Chacko42

[RyanGates]

Hi Chacko,

I did the reschedule for the updates and it did not seem to work for us. Support got on the line and discovered an

error. c:698): Bad update information on disk2018-04-13 10:30:29.609 -0400 No new Content content available for download.

She said because of the Bad Update information on Disk error she restarted our "debug software restart process device-server", which fixed the issue for us. We scheduled the update to run in 5 minutes and it grabbed the update successfully on its own. 

Hope yours was fixed with the solution you posted. However, if it’s not. Try the solution that she gave us. She said it wouldn't impact production but I would check with support first if you think otherwise.

Thanks,

RG