Order to reboot devices in HA pair (passive)

[bino150]

We need to reboot our firewall due to some issues related to the traffic logging not working.  We have already attempted debug software restart log-receiver, syncing the devices etc and none of them have resolved the issue. We are pretty new to the device and have never had to reboot them. We have two PA-500's in an HA pair config. The backup is passive. It was suggested we reboot both devices.

What is the proper order if we intend to reboot both devices? If we reboot the main firewall will it initiate a reboot of the backup device or do we need to reboot each device separately?

[HULK]

Hello Bino,

Before re-start the PAN firewall, i would request you to follow  below mentioned steps:

Question-1:

>show logging-status 

Check: Last Log fwded and Last SeqNo. fwded counters. Make sure dates are showing correctly and sequence number is incrementing.

>debug log-receiver statistics

Check Log incoming rate and Log written rate are incrementing. Make sure below mentioned counters are not incrementing rapidly:

Log Forward discarded (queue full) count: 0  >>>>>>

Log Forward discarded (send error) count: 0  >>>>>>

Apply below mentioned command as per the sequence:

>debug software trace log-receiver

>debug software trace management-server

>debug software restart log-receiver

if no change still;

>debug software restart management-server 

Once you will restart the management-server process, it will take some time to come up, you will lose cli access for a few minutes and it will not impact to the production traffic through data-plane.

> debug dataplane pool statistics  >>>>>>>>> Verify Software pools are not depleted

> show system software status | match logrcvr  ( Restart may be required if not running/stopped)

Question-2: If the PA-500 HA pair is in a production environment, i would suggest you not to restart both firewalls at the same time. First restart the Active firewall, so the Secondary will become Active ( for the time being) and it will start passing production traffic. Once, it will become up, you may restart the second firewall.

Please let me know, if you have any other questions or concerns I can help address regarding this issue.

Thanks

[hshah]

Hi Bino,

Make sure partition is not full, that might be impacting logging.

Execute "show system disk-space" command.

admin@PA-500-Gia(active)> show system disk-space

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             3.8G  1.4G  2.3G  38% /

/dev/sda5             7.6G  3.8G  3.4G  53% /opt/pancfg

/dev/sda6             3.8G  2.1G  1.6G  58% /opt/panrepo

tmpfs                 991M   67M  924M   7% /dev/shm

/dev/sda8             125G  2.3G  116G   2% /opt/panlogs--------------> Make sure this has space

Regards,

Hardik Shah

[hshah]

Hi Bino,

In HA, only active one does logging, and not passive unit. Than what is purpose of rebooting passive.

Passive might be able to do logging.

If you really want to reboot Active unit than follow bellow mentioned steps.

1. Suspend Active, now passive will take over.

request high-availability state suspend

2. Check if passive[New active] does logging for traffic logs.

3. If passive[New Active] does logging than reboot is not required.

4. At this point you can reboot active[new Passive] unit as its not passing traffic.

5. If passive[New Active] doesnt do logging than follow the same process.

Make sure you do not reboot both the boxes same time, at a time atleast one box should be passing traffic. That way you can avoid any kind of potential outage.

Regards,

Hardik Shah

[bino150]

Thank you for the info guys.

Here is the system disc space. it doesn't look too full

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda3             3.8G  1.7G  1.9G  47% /

/dev/sda5             7.6G  2.9G  4.4G  40% /opt/pancfg

/dev/sda6             3.8G  1.2G  2.5G  33% /opt/panrepo

tmpfs                 963M   67M  896M   7% /dev/shm

/dev/sda8             125G   13G  106G  11% /opt/panlogs

Here is the logging status>

--------------------------------------------------------------------------------

---------------------------------------------

      Type      Last Log Created        Last Log Fwded       Last Seq Num Fwded

Last Seq Num Acked         Total Logs Fwded

--------------------------------------------------------------------------------

---------------------------------------------

> CMS 0

        Not Sending to CMS 0

> CMS 1

        Not Sending to CMS 1

>Log Collector

        Not Sending to Log Collector

Debug log-receiver statistics

admin@PA-500(active)> debug log-receiver statistics

Server error : An error occured. See dagger.log for information.

admin@PA-500(active)>

I've tried restart log-receiver several and restart management-server times a few days ago. it did get logging going for system etc but no effect for the traffic log.

>debug software trace log-receiver

>debug software trace management-server

>debug software restart log-receiver

if no change still;

>debug software restart management-server

This all relates back to this post which HULK helped me with.Logging stopped in Pan OS GUI

I did do show ntp as well

NTP state:

    NTP synched to 0.north-america.pool.ntp.org

    NTP server 0.north-america.pool.ntp.org connected: True

    NTP server 1.north-america.pool.ntp.org connected: True

I think a reboot is still needed to resolve this problem.

[HULK]

Hello Bino,

Instead of rebooting the PAN firewall, i would recommend you to open a ticket with PAN support and let them investigate this. So, you can prevent any future occurrence as well.

Thanks

[bino150]

Thanks Hulk.

[craymond]

As HULK said, you should definitely call support first. Before you suspend the active unit, it is recommended that you disable preemptive under election settings to prevent unwanted elections. Disabling the preempt configuration change must be committed on BOTH peers, and once completed, re-enabling must be committed on both peers. The setting is located in High Availability -> General Tab.