topic Re: Order to reboot devices in HA pair (passive) in General Topics

Order to reboot devices in HA pair (passive)

bino150 — Wed, 16 Jul 2014 18:01:23 GMT

We need to reboot our firewall due to some issues related to the traffic logging not working. We have already attempted debug software restart log-receiver, syncing the devices etc and none of them have resolved the issue. We are pretty new to the device and have never had to reboot them. We have two PA-500's in an HA pair config. The backup is passive. It was suggested we reboot both devices.

What is the proper order if we intend to reboot both devices? If we reboot the main firewall will it initiate a reboot of the backup device or do we need to reboot each device separately?

Re: Order to reboot devices in HA pair (passive)

HULK — Wed, 16 Jul 2014 19:28:11 GMT

Hello Bino,

Before re-start the PAN firewall, i would request you to follow below mentioned steps:

Question-1:

>show logging-status

Check: Last Log fwded and Last SeqNo. fwded counters. Make sure dates are showing correctly and sequence number is incrementing.

>debug log-receiver statistics

Check Log incoming rate and Log written rate are incrementing. Make sure below mentioned counters are not incrementing rapidly:

Log Forward discarded (queue full) count: 0 >>>>>>

Log Forward discarded (send error) count: 0 >>>>>>

Apply below mentioned command as per the sequence:

>debug software trace log-receiver

>debug software trace management-server

>debug software restart log-receiver

if no change still;

>debug software restart management-server

Once you will restart the management-server process, it will take some time to come up, you will lose cli access for a few minutes and it will not impact to the production traffic through data-plane.

> debug dataplane pool statistics >>>>>>>>> Verify Software pools are not depleted

> show system software status | match logrcvr ( Restart may be required if not running/stopped)

Question-2: If the PA-500 HA pair is in a production environment, i would suggest you not to restart both firewalls at the same time. First restart the Active firewall, so the Secondary will become Active ( for the time being) and it will start passing production traffic. Once, it will become up, you may restart the second firewall.

Please let me know, if you have any other questions or concerns I can help address regarding this issue.

Thanks

Re: Order to reboot devices in HA pair (passive)

hshah — Wed, 16 Jul 2014 20:52:21 GMT

Hi Bino,

Make sure partition is not full, that might be impacting logging.

Execute "show system disk-space" command.

admin@PA-500-Gia(active)> show system disk-space

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 3.8G 1.4G 2.3G 38% /

/dev/sda5 7.6G 3.8G 3.4G 53% /opt/pancfg

/dev/sda6 3.8G 2.1G 1.6G 58% /opt/panrepo

tmpfs 991M 67M 924M 7% /dev/shm

/dev/sda8 125G 2.3G 116G 2% /opt/panlogs--------------> Make sure this has space

Regards,

Hardik Shah

Re: Order to reboot devices in HA pair (passive)

hshah — Wed, 16 Jul 2014 20:57:50 GMT

Hi Bino,

In HA, only active one does logging, and not passive unit. Than what is purpose of rebooting passive.

Passive might be able to do logging.

If you really want to reboot Active unit than follow bellow mentioned steps.

1. Suspend Active, now passive will take over.

request high-availability state suspend

2. Check if passive[New active] does logging for traffic logs.

3. If passive[New Active] does logging than reboot is not required.

4. At this point you can reboot active[new Passive] unit as its not passing traffic.

5. If passive[New Active] doesnt do logging than follow the same process.

Make sure you do not reboot both the boxes same time, at a time atleast one box should be passing traffic. That way you can avoid any kind of potential outage.

Regards,

Hardik Shah

Re: Order to reboot devices in HA pair (passive)

bino150 — Wed, 16 Jul 2014 22:38:53 GMT

Thank you for the info guys.

Here is the system disc space. it doesn't look too full

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 3.8G 1.7G 1.9G 47% /

/dev/sda5 7.6G 2.9G 4.4G 40% /opt/pancfg

/dev/sda6 3.8G 1.2G 2.5G 33% /opt/panrepo

tmpfs 963M 67M 896M 7% /dev/shm

/dev/sda8 125G 13G 106G 11% /opt/panlogs

Here is the logging status>

--------------------------------------------------------------------------------

---------------------------------------------

Type Last Log Created Last Log Fwded Last Seq Num Fwded

Last Seq Num Acked Total Logs Fwded

--------------------------------------------------------------------------------

---------------------------------------------

> CMS 0

Not Sending to CMS 0

> CMS 1

Not Sending to CMS 1

>Log Collector

Not Sending to Log Collector

Debug log-receiver statistics

admin@PA-500(active)> debug log-receiver statistics

Server error : An error occured. See dagger.log for information.

admin@PA-500(active)>

I've tried restart log-receiver several and restart management-server times a few days ago. it did get logging going for system etc but no effect for the traffic log.

>debug software trace log-receiver

>debug software trace management-server

>debug software restart log-receiver

if no change still;

>debug software restart management-server

This all relates back to this post which HULK helped me with.Logging stopped in Pan OS GUI

I did do show ntp as well

NTP state:

NTP synched to 0.north-america.pool.ntp.org

NTP server 0.north-america.pool.ntp.org connected: True

NTP server 1.north-america.pool.ntp.org connected: True

I think a reboot is still needed to resolve this problem.

Re: Order to reboot devices in HA pair (passive)

HULK — Thu, 17 Jul 2014 07:27:46 GMT

Hello Bino,

Instead of rebooting the PAN firewall, i would recommend you to open a ticket with PAN support and let them investigate this. So, you can prevent any future occurrence as well.

Thanks

Re: Order to reboot devices in HA pair (passive)

bino150 — Thu, 17 Jul 2014 21:22:12 GMT

Thanks Hulk.

Re: Order to reboot devices in HA pair (passive)

craymond — Fri, 18 Jul 2014 16:59:07 GMT

As HULK said, you should definitely call support first. Before you suspend the active unit, it is recommended that you disable preemptive under election settings to prevent unwanted elections. Disabling the preempt configuration change must be committed on BOTH peers, and once completed, re-enabling must be committed on both peers. The setting is located in High Availability -> General Tab.