VPN client certificates rejected until firewall reboot

[fhewiufhwefhwe]

I had to reboot my firewall this morning because it erroneously rejected client certificates required by a VPN.

Firewall system logs show critical event "Out of memory condition detected, kill process 3" at 4:06am

I had the exact same issue on May 5th as well (and reporting to PA) where Clients getting VPN certificate errors despite being nowhere near expiration and reinstalling certifications

Is anyone aware of a fix?

[BPry]

@fhewiufhwefhwe,

I've ran into this a few times with 10.0 throughout various releases and haven't gotten an actual direct answer from support. I'd keep reporting it, because it's definitely a bug somewhere that they just don't appear to have enough data to track down yet.  

[Remo · Accepted solution]

I also have seen this issue. Clients were not able to connect and they were presented with a message that a valid certificste is required. I also saw the out of memory logs. After that I installed PAN-OS 9.1.10 which has quite a few fixes for something that could result in this problem. So far the error did not happen again.

[fhewiufhwefhwe]

Are either of you running in HA Pair?  I am wondering whether or not that might mitigate the issue in active-passive and/or active-active until there is a bug fix.  Both times this issue occurred early morning, and fortunately only two people were in the office by then.

[Remo]

I had the issue in a HA pair (active-passive). Actually we have more than 10 other firewall HA pairs where we use global protect, but so far (luckily) the issue only happened on one of them ...

[fhewiufhwefhwe]

Got it.  So the passive firewall took over while you rebooted the problematic active firewall, and users didn't have downtime during the reboot.  Is that correct?  How much time did it take to configure active-passive mode for the first time?

[Remo]

As long as you immediately reboot the firewall after the OOM systemlog, then yes you will be able to reduce the downtime to almost 0. Otherwise there will still be a timeframe where users are not able to connect.

Setting up a HA pair on the firewallside is quite easy to do. The walkthrough with a step by step manual you can find here: https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/high-availability/set-up-activepassive-ha/...

Depending on thw network setup you need to change some things there too.

What PAN-OS version do you currently run on this firewall?

[fhewiufhwefhwe]

9.1.9

I tried upgrading to 10.0 a couple of times last year, but found it too buggy at the time.  Not sure if it stable enough to run production now, but I will likely wait at least a few more weeks before considering an upgrade.

[Remo]

I think you should consider an update to 9.1.10. Maybe the situation gets also better for you and maybe the issue is already completely resolved in this version

[fhewiufhwefhwe]

Agreed.  I'll likely try it this weekend

[Remo]

Hi @fhewiufhwefhwe 

did you do the update to 9.1.10 and if so, did the problem happen again since then?

[fhewiufhwefhwe]

Updated, but the issue occurred between 30 and 50 days uptime after a memory error.  Within waiting two months or a reocurrence, I have no way to confirm that the issue has been fixed.  The release notes did not mention a similar issue.

[asangra]

That could be an issue with time sync b/w MP and DP. You may need to check with NTP servers, if any. Reboot makes both MP and DP clock in sync and for more info follow below.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Clh4CAC 

PAN-160744

https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-release-notes/pan-os-9-1-addressed-issues/pan-os-9-1-9-addressed-issues.html