topic Re: Pa-2020 restart unexpected in General Topics

Pa-2020 restart unexpected

Oskare_o — Thu, 30 Oct 2014 19:28:22 GMT

Hello

My Pa-2020 restart unexpectedly with no reason aparently. I'm attaching the log so any help me is welcome.

Thanks!

Log:

2014/10/30 11:04:56 info general general 0 Connection to Update server: updates.paloaltonetworks.com completed successfu

lly, initiated by 192.168.48.183

!----------- System Restart at 11:59

2014/10/30 11:59:24 info general general 0 Management server started. Running version 5.0.9

2014/10/30 11:59:24 info general general 0 VPN Disable mode = off

2014/10/30 11:59:26 info ntpd restart 0 NTP restart synchronization performed

2014/10/30 11:59:26 high general system- 1 The system is starting up.

2014/10/30 11:59:26 info satd satd-da 0 SATD daemon is initializing.

2014/10/30 11:59:26 info vpn keymgr- 0 KEYMGR daemon is initializing.

2014/10/30 11:59:26 info ras rasmgr- 0 RASMGR daemon is initializing.

2014/10/30 11:59:26 info url-fil url-eng 0 BrightCloud engine started.

2014/10/30 11:59:26 info vpn ike-dae 0 IKE daemon is initializing.

2014/10/30 11:59:26 info vpn keymgr- 0 KEYMGR sync all IPSec SA to IKE daemon started.

2014/10/30 11:59:26 info routing routed- 0 Route daemon is initializing.

2014/10/30 11:59:26 info vpn keymgr- 0 KEYMGR daemon is ready.

2014/10/30 11:59:26 info satd satd-da 0 SATD daemon is ready.

2014/10/30 11:59:26 info vpn keymgr- 0 KEYMGR sync all IPSec SA to IKE daemon exit.

2014/10/30 11:59:26 info ras rasmgr- 0 RASMGR daemon is ready.

2014/10/30 11:59:26 info vpn ike-dae 0 IKE daemon is ready.

2014/10/30 11:59:26 info sslmgr sslmgr- 0 SSLMGR daemon is ready.

2014/10/30 11:59:26 info routing routed- 0 Route daemon is ready.

2014/10/30 11:59:26 info vpn keymgr- 0 KEYMGR sync all IPSec SA to Flow no longer needed.

Re: Pa-2020 restart unexpected

hshah — Thu, 30 Oct 2014 19:37:16 GMT

Please provide us output for "show system files".

and show system info.

Re: Pa-2020 restart unexpected

HULK — Thu, 30 Oct 2014 19:37:55 GMT

Hello Oskare_o,

I hope you have attached PAN SYSTEM logs here. Is there any log entries just before the incident happened. (11:59) ?

Thanks

Re: Pa-2020 restart unexpected

Oskare_o — Thu, 30 Oct 2014 19:58:30 GMT

The file crash info is empty or i don't know how to export it. I do the following:

admin@PA-2020> show system files

/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Oct 30 11:48 crashinfo

/var/cores/crashinfo:
total 16K
-rw-r--r-- 1 root root 16K Oct 30 11:48 kernel_panic_0

/opt/dpfs/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Mar 7 2014 crashinfo

/opt/dpfs/var/cores/crashinfo:
total 0
admin@PA-2020> tftp export core-file management-plane from crashinfo to 172.20.103.6
mode set to octet
Connected to 172.20.103.6 (172.20.103.6), port 69
putting /var/cores/crashinfo to 172.20.103.6:crashinfo [octet]

admin@PA-2020> tftp export core-file data-plane from crashinfo to 172.20.103.6

mode set to octet

Connected to 172.20.103.6 (172.20.103.6), port 69

putting /opt/dpfs/var/cores/crashinfo to 172.20.103.6:crashinfo [octet]

But the files ares empty, 0Kb.

This is the show system info output:

hostname: PA-2020
ip-address: 192.168.48.183
netmask: 255.255.255.0
default-gateway: 192.168.48.254
ipv6-address:
ipv6-link-local-address: fe80::21b:17ff:fe79:b700/64
ipv6-default-gateway:
mac-address: 00:1b:17:79:b7:00
time: Thu Oct 30 13:53:16 2014
uptime: 0 days, 1:48:54
family: 2000
model: PA-2020
serial: 0004C104280
sw-version: 5.0.9
global-protect-client-package-version: 0.0.0
app-version: 466-2435
app-release-date: 2014/10/28 20:28:09
av-version: 1401-1873
av-release-date: 2014/10/24 04:00:01
threat-version: 466-2435
threat-release-date: 2014/10/28 20:28:09
wildfire-version: 0
wildfire-release-date: unknown
url-filtering-version: 4111
global-protect-datafile-version: 0
global-protect-datafile-release-date: unknown
logdb-version: 5.0.2
platform-family: 2000
logger_mode: False
vpn-disable-mode: off
operational-mode: normal
multi-vsys: off

HULK

Before that logs there is this:

2014/10/30 08:03:52 info userid connect 0 ldap cfg Group01 connected to server 192.168.48.212:3268, initiated by: 192.1

68.48.183

2014/10/30 09:03:54 info userid connect 0 ldap cfg Group01 connected to server 192.168.48.212:3268, initiated by: 192.1

68.48.183

2014/10/30 10:03:56 info userid connect 0 ldap cfg Group01 connected to server 192.168.48.212:3268, initiated by: 192.1

68.48.183

2014/10/30 10:43:30 info general auth-su 0 User 'oortiz' authenticated. From: 172.20.103.6.

2014/10/30 10:43:31 info general general 0 User oortiz logged in via Web from 172.20.103.6 using https

2014/10/30 10:43:31 info general general 0 Session for user lolivares via Web from 172.20.103.3 timed out

2014/10/30 11:03:58 info userid connect 0 ldap cfg Group01 connected to server 192.168.48.212:3268, initiated by: 192.1

68.48.183

2014/10/30 11:04:56 info general general 0 Connection to Update server: updates.paloaltonetworks.com completed successfu

lly, initiated by 192.168.48.183

2014/10/30 11:59:24 info general general 0 Management server started. Running version 5.0.9

2014/10/30 11:59:24 info general general 0 VPN Disable mode = off

2014/10/30 11:59:26 info ntpd restart 0 NTP restart synchronization performed

Thank you alot!

Re: Pa-2020 restart unexpected

HULK — Thu, 30 Oct 2014 20:03:07 GMT

Thanks for your update. It looks like, there is no relevant SYSTEM logs during the incident. It would be better to contact support and let them analyze the tech-support file for root cause.

Thanks

Re: Pa-2020 restart unexpected

hshah — Thu, 30 Oct 2014 20:06:04 GMT

Hi Oskare,

Firewall generated kernal panic at the time of reboot, which means its hitting one or other bug.

Please provide me output for following command. Those commands are used to read crash info files.

1. less mp-backtrace kernel_panic_0

2. less mp-backtrace kernel_panic_0 >>> Most likely you will get error with one of this two commands.

Regards,

Hardik Shah

Re: Pa-2020 restart unexpected

Oskare_o — Thu, 30 Oct 2014 22:34:31 GMT

hshah

I shared to you a file with less mp-backtrace kernel_panic_0

Thanks you!!!

Re: Pa-2020 restart unexpected

hshah — Thu, 30 Oct 2014 22:35:26 GMT

Hi OSkare,

Thanks for following file, I will update you soon

https://live.paloaltonetworks.com/docs/DOC-8259)

Regards,

Hardik Shah

Re: Pa-2020 restart unexpected

hshah — Thu, 30 Oct 2014 22:40:35 GMT

Hi Oskare,

I see following error in crash info.

Applying R_ERR on DMA activate FIS errata fix

This may or may not be a HDD RMA, I would suggest you to contact TAC and verify the same.

Regards,

Hardik Shah

Re: Pa-2020 restart unexpected

HULK — Thu, 30 Oct 2014 22:43:26 GMT

Hello Oskare_o,

From the back trace it looks like the firewall was unable to read filesystem from the HDD drive during that time. It is showing the ATA link error and to recover from this situation it got rebooted automatically.

3>ata1.00: failed command: READ DMA >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ATA link error

<3>ata1.00: cmd c8/00:08:f1:44:25/00:00:00:00:00/ec tag 0 dma 4096 in

<3> res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x56 (ATA bus error)

<3>ata1.00: status: { DRDY }

<6>ata1: hard resetting link

<3>ata1: failed to resume link (SControl FFFFFFFF)

<6>ata1: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF) >>>>>>>>>>>>>>>>>>>>>> link to the HDD was down

<6>ata1: hard resetting link

<3>ata1: failed to resume link (SControl FFFFFFFF)

<6>ata1: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)

<6>ata1: hard resetting link

<3>ata1: failed to resume link (SControl FFFFFFFF)

<6>ata1: SATA link down (SStatus FFFFFFFF SControl FFFFFFFF)

<4>ata1.00: disabled

<6>sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08

<6>sd 0:0:0:0: [sda] Sense Key : 0xb [current] [descriptor]

<4>Descriptor sense data with sense descriptors (in hex):

<6> 72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00

<6> 00 00 00 00

<6>sd 0:0:0:0: [sda] ASC=0x0 ASCQ=0x0

<6>sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 0c 25 44 f1 00 00 08 00

<3>end_request: I/O error, dev sda, sector 203769073

<6>ata1: EH complete

<6>ata1.00: detaching (SCSI 0:0:0:0)

<1>Read-error on swap-device (8:0:44701239)

<1>Read-error on swap-device (8:0:44841087)

<1>Read-error on swap-device (8:0:44841103)

<1>Read-error on swap-device (8:0:44841111)

<1>Read-error on swap-device (8:0:44841119)

<1>Read-error on swap-device (8:0:44241407)

<1>Read-error on swap-device (8:0:44241415)

<1>Read-error on swap-device (8:0:44241423)

<1>Read-error on swap-device (8:0:44241431)

<1>Read-error on swap-device (8:0:44241447)

<1>Read-error on swap-device (8:0:44241455)

<1>Read-error on swap-device (8:0:44241463)

<1>Read-error on swap-device (8:0:44193367)

<1>Read-error on swap-device (8:0:44193375)

<1>Read-error on swap-device (8:0:44193383)

<1>Read-error on swap-device (8:0:44193399)

I would suggest you to contact PAN support and you may need to replace the HDD on this FW..

Hope this helps.

Thanks

Re: Pa-2020 restart unexpected

Oskare_o — Fri, 31 Oct 2014 00:26:26 GMT

Thank for your help, I'm going to contact my suport partner,

Thank you very much!!

Saludos