PA-220 Login issue

[R.Moth]

Hi Everyone

We have a load of PA-220's in HA at variuos different sites, after a set period of time the devices stop allowing use to login.  Does not matter what method of Auth we try, it just does not work.  To compond the problem, if we do not resolve this issue eariler enough both devices at the site go offline taking the whole site down.  The issue started around 18 months to 2 years ago.  When we first had the issue we opened several support case with Palo Alto and ended up breaking several devices.  Those devices then had to be replaced by Palo Alto.  Eventually with Palo Alto support we found a workaround.  Here are the steps to get the firewalls back online.

1. Connect with console cable to backup firewall

2. Power cycle the device

3. Go into Maintenance Mode

4. Re-install PANOS.

5. Reboot

6. Once the device is back online and the site is reachable again perform the same steps on the primary firewall.

So, we do have a workaround but that requires someone to be on site with a laptop that can give us access to the devices, which is not always an easy task.  All the PA-220's are located in off site data centers.

I've only been working with Palo Alto devices for about 2.5 years, but my hunch is that 12 of the drives is filling up and causing the problem.  The issue is once we do the reinstall you can see what was the problem.

One of our other senior engineers had reported that he had seen on a PA forum they issue could be linked to a memory issue and restarting a process on both PA's before pushing a configuration update out to the devices from Pnaorama.

Can anyone suggests anything that we can monitor on the devices like memory or storage to see if the issues are related to either memory or storage?

Richard

[YEmreSeven]

Hi @R.Moth, 

Actually most of problem's root cause related with newer PANOS versions is not giving good performance with PA-220. So I can suggest stay in lower major versions as much as possible.

In my experience with PA-220, problems are starting after when root partition usage over %95. Hardware reboot can be decrease a bit, but it will increase eventually. As I understand from your re-installing OS is also helping to clean whole disk. 

You can monitor your root partition too. And if you see the same behaviour, you can try these cleanup commands, 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaJCAS

Emre

[R.Moth]

Hi Emre

Thank you for your reply, Palo Alto have feed the same information back to use yesterday.  I've setup some alerting in our monitoring tool to raise an Alert when the root partition goes over 91%.  At the moment they are all just under 90%.  I did run some checks on 1 device yesterday, but if that is the problem unless Palo Alto come up with a fix to free up more disk space we are always going to hit the problem after a period of time.

Richard

[YEmreSeven]

Hi Richard,

Actually root partition issue for PA-220 looks "solved" at 11.1.5. 

"PAN-258166 (PA-220 firewalls only) Fixed an issue where the root partition frequently reached 100%."

https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-release-notes/pan-os-11-1-5-known-and-addressed... 

But I couldn't try because 11.1 is not supported for PA-220.

Emre

[R.Moth]

Hi Emre

That's interesting, but as you say 11.1 is not support on the PA-220.  Don't understand why they fix the issue in 11.1 when that version is not suppoted.  I might raise a ticket with Palo Alto and see what they say.

I'm going to raise a  ticket with Palo Alto and see what the stance is on this.  Just wondering if the 220's are not offically supported in 11.1 but will work if installed.

Thank you for that information.

Richard

[OtakarKlier]

Hello,

When you mention it doesnt allow login, are you referring to the GUI or CLI?

Regards,