topic Re: Logging stopped in Pan OS GUI in General Topics

Logging stopped in Pan OS GUI

bino150 — Tue, 08 Jul 2014 21:05:20 GMT

Hi all,

We had an issue today where we noticed the logging data stopped displaying in the Pan OS GUI (PA-500).. One thing we noticed is after committing a config change, it looks like the firewall was in the process of doing a sync with the backup device. I assume to resolve this problem we need to restart the dataplane? I'm surprised it allowed the commit to occur if a sync was in progress.

Bryan

Re: Logging stopped in Pan OS GUI

pulukas — Tue, 08 Jul 2014 21:09:21 GMT

Logging is handled by the log receiver.

> debug software restart log-receiver

You can check statistics here to see which are not working.

>debug log-receiver statistics

Re: Logging stopped in Pan OS GUI

HULK — Tue, 08 Jul 2014 21:17:35 GMT

Hello Bino,

Could you please check below mentioned command:

> show logging-status >>>>>>>>> Check last forwarded logs date and time

> debug log-receiver statistics ------ check if below mentioned counters are incrementing

Log Forward discarded (queue full) count: 0 >>>>

Log Forward discarded (send error) count: 0 >>>>

>debug software restart log-receiver

if no change still;

>debug software restart management-server ---- after applying this command, wait for a few minutes. It will log you out from CLI and GUI.

Thanks

Re: Logging stopped in Pan OS GUI

bino150 — Tue, 08 Jul 2014 21:41:54 GMT

Thanks Guys,

I assume the debug software restart log-receiver can be done while the firewall is up? We were thinking of waiting until after business hours do this. Any idea what would cause this issue? I assume maybe it was committing a change at the same time as the sync.

Re: Logging stopped in Pan OS GUI

HULK — Tue, 08 Jul 2014 21:55:51 GMT

Hello Bino,

Yes, you can run those 2 commands while the FW is UP. It would not impact to the data-plane traffic ( user's traffic through PAN firewall), because daemons are running on Management-plane. For safer side, you may restart log-receiver and management server process after the business hrs.

Thanks

Re: Logging stopped in Pan OS GUI

bino150 — Tue, 08 Jul 2014 22:31:17 GMT

Thanks Hulk

Re: Logging stopped in Pan OS GUI

HULK — Tue, 08 Jul 2014 22:43:28 GMT

You are welcome.

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 00:07:26 GMT

Hulk we still seem to have an issue. I ran both debug software restart log-receiver which did not resolve the issue. Same with debug software restart management-server.

if I run debug log-receiver statistics I get the following error see dagger log. Do I need to do this plugged in to the cli port directly. I'm doign this via telnet from my workstation on the LAN.

Re: Logging stopped in Pan OS GUI

HULK — Wed, 09 Jul 2014 00:12:10 GMT

Hello Bino,

If you restart the management-server daemon, you have to wait for a few minutes. It will automatically log out from CLI (SSH), since SSH/web-UI is managed by mgmt-server process. So, please re-login into the PAN firewall and then check with CLI command >debug log-receiver statistics

Thanks

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 00:17:58 GMT

Just logged in again and the error is still coming up. its been a few minutes now,

debug log-receiver statisitcs

Server error: An error occurred. See dagger.log for information.

Re: Logging stopped in Pan OS GUI

HULK — Wed, 09 Jul 2014 00:28:10 GMT

Could you please check CLI command o/p of > show ntp

It should show the NTP server, it's connected.

Related DOC:

Error in NTP Sync Status Display

Let me know the result.

Thanks

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 00:29:27 GMT

Think I need to contact support for this. Thanks for the help

Re: Logging stopped in Pan OS GUI

HULK — Wed, 09 Jul 2014 00:33:14 GMT

The mentioned error messages are related to NTP. Hence, could you please check the status of the NTP server on your FW.

Server error: An error occurred. See dagger.log for information

Thanks

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 00:37:11 GMT

Low and behold the log information is coming through now. I'm still getting the Server error: An error occurred. See dagger.log for information running debug log-receiver statistics from CLI though.

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 00:39:16 GMT

Interesting I am showing connected false for NTP

admin@PA-500(active)> show ntp

NTP state:

NTP synched to LOCAL

NTP server 0.north-america.pool.ntp.org connected: False

NTP server 1.north-america.pool.ntp.org connected: False

Re: Logging stopped in Pan OS GUI

HULK — Wed, 09 Jul 2014 00:47:12 GMT

Yes, NTP is the issue behind the error message "Server error: An error occurred. See dagger.log for information". Please check above mentioned KB DOC for the same.

Hope this helps.

Thanks

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 00:51:11 GMT

Hulk thanks so much for the help. The NTP server re-synched and now its showing true so it looks good now.

Re: Logging stopped in Pan OS GUI

bino150 — Wed, 09 Jul 2014 03:51:13 GMT

I think I spoke too soon. We are getting system log info etc. I tried checking just now and I notice the traffic log is not updating or displaying in the monitor. NTP seems to be fine.

I ran debug log-receiver statistics and the error "An error occured. See dagger.log for information." still appears.

I ran tail output after running debug log-receiver statistics and I see the following

KeyError: 'sw.logrcvr.runtime.opcmd'

admin@PA-500(active)> tail mp-log dagger.log

2014-07-08 20:44:23.567 -0700 OPCMD: handler "logrcvr_handler.debug_stats"

Traceback (most recent call last):

File "/usr/share/dagger/logrcvr.py", line 60, in debug_stats

resp_node = self.sdb.modify(name=op_node_name, object=op_node, timeout=60)

KeyError: 'sw.logrcvr.runtime.opcmd'

2014-07-08 20:45:11.212 -0700 OPCMD: handler "logrcvr_handler.debug_stats"

Traceback (most recent call last):

File "/usr/share/dagger/logrcvr.py", line 60, in debug_stats

resp_node = self.sdb.modify(name=op_node_name, object=op_node, timeout=60)

KeyError: 'sw.logrcvr.runtime.opcmd'

Is it possible I need to let the devices re-sync. Is there a command to manually force the devices to resync?

Re: Logging stopped in Pan OS GUI

pulukas — Wed, 09 Jul 2014 23:24:22 GMT

You can force the sync from the secondary cli using this command.

request high-availability sync-to-remote running-config

I doubt this is the issue but it can confirm what you want to confirm.

I would try to schedule a reboot of both nodes in turn after a sync confirmed. If the issue is still present I would open a case.

Re: Logging stopped in Pan OS GUI

bino150 — Fri, 11 Jul 2014 21:22:00 GMT

Thank you Steven. (and Hulk). You have been very helpful. We intend to reboot the devices. As you mentioned the sync did not resolve the issue. Very odd indeed. All other logs system logs etc are working fine except traffic and some of the threat logs.