SYSTEM ALERT : high : all_pktproc_2: exiting because missed too many heartbeats

[avi_goldstain]

i recive this error few times a week

subject : SYSTEM ALERT : high : all_pktproc_2: exiting because missed too many heartbeats

domain: 1
receive_time: 2014/11/26 13:18:11
serial: 009401009996
seqno: 43452
actionflags: 0x0
type: SYSTEM
subtype: general
config_ver: 0
time_generated: 2014/11/26 13:18:11
vsys:
eventid: general
object:
fmt: 0
id: 0
module: general
severity: high
opaque: all_pktproc_2: exiting because missed too many heartbeats

please help

[panos]

can you type output of   show system software status

[gbogojevic]

Could you please send output of the "show system files" - If you have some core dump files generated, you should open a case with Palo Alto TAC

[avi_goldstain]

here is the output off show system software status

Slot 0, Role mp

----------------------------------------

Type     Name                 State    Info

Group    all                  running

Process  l3svc                running  (pid: 4835)

Process  logrcvr              running  (pid: 2810)

Process  masterd              running  (pid: 2298)

Process  mgmtsrvr             running  (pid: 2791)

Process  monitor              running  (pid: 2326)

Process  pppoe                running  (pid: 2819)

Process  rasmgr               running  (pid: 2811)

Process  routed               running  (pid: 2820)

Process  satd                 running  (pid: 2815)

Process  snmpd                running  (pid: 7113)

Process  sshd                 running  (pid: 2661)

Process  sslmgr               running  (pid: 2816)

Process  sslvpn               running  (pid: 4840)

Process  sysd                 running  (pid: 2310)

Process  sysdagent            running  (pid: 2330)

Process  useridd              running  (pid: 2769)

Process  varrcvr              running  (pid: 2813)

Process  websrvr              running  (pid: 24812)

Slot 1, Role dp0

----------------------------------------

Type     Name           State    Info

Group    all            running

Group    gdb            running

Group    platform       running

Group    supervisor     running

Group    tasks          running

Process  all_pktproc_2  running  (pid: 14673)

Process  brdagent       running  (pid: 862)

Process  comm           running  (pid: 881)

Process  dha            running  (pid: 882)

Process  ehmon          running  (pid: 864)

Process  flow_mgmt      running  (pid: 885)

Process  masterd        running  (pid: 834)

Process  monitor        running  (pid: 891)

Process  mprelay        running  (pid: 880)

Process  ntp            running  (pid: 13767)

Process  pktproc_n_log  running  (pid: 14614)

Process  sysdagent      running  (pid: 863)

here is output off show system software status

-rw-r--r-- 1 root root 147 Nov  4 15:30 all_pktproc_2_6.0.5_0.pcap

-rw-rw-rw- 1 root root 22K Nov  4 15:31 all_pktproc_2_6.0.5_0.info

-rw-r--r-- 1 root root 121 Nov  6 15:12 all_pktproc_2_6.0.5_1.pcap

-rw-rw-rw- 1 root root 21K Nov  6 15:12 all_pktproc_2_6.0.5_1.info

-rw-r--r-- 1 root root 121 Nov  6 15:13 pktproc_n_log_6.0.5_0.pcap

-rw-rw-rw- 1 root root 21K Nov  6 15:13 pktproc_n_log_6.0.5_0.info

-rw-r--r-- 1 root root 147 Nov 12 15:56 all_pktproc_2_6.0.5_2.pcap

-rw-rw-rw- 1 root root 21K Nov 12 15:57 all_pktproc_2_6.0.5_2.info

-rw-r--r-- 1 root root 147 Nov 19 10:42 pktproc_n_log_6.0.5_1.pcap

-rw-rw-rw- 1 root root 21K Nov 19 10:42 pktproc_n_log_6.0.5_1.info

-rw-r--r-- 1 root root 147 Nov 19 10:43 all_pktproc_2_6.0.5_3.pcap

-rw-rw-rw- 1 root root 21K Nov 19 10:43 all_pktproc_2_6.0.5_3.info

-rw-r--r-- 1 root root 147 Nov 20 14:04 pktproc_n_log_6.0.5_2.pcap

-rw-rw-rw- 1 root root 22K Nov 20 14:05 pktproc_n_log_6.0.5_2.info

-rw-r--r-- 1 root root 147 Nov 23 08:58 pktproc_n_log_6.0.5_3.pcap

-rw-rw-rw- 1 root root 21K Nov 23 08:59 pktproc_n_log_6.0.5_3.info

-rw-r--r-- 1 root root 147 Nov 26 13:16 all_pktproc_2_6.0.5_4.pcap

-rw-rw-rw- 1 root root 21K Nov 26 13:16 all_pktproc_2_6.0.5_4.info

-rw-r--r-- 1 root root 147 Nov 26 13:17 pktproc_n_log_6.0.5_4.pcap

-rw-rw-rw- 1 root root 22K Nov 26 13:17 pktproc_n_log_6.0.5_4.info

-rw-r--r-- 1 root root 147 Nov 26 13:18 all_pktproc_2_6.0.5_5.pcap

-rw-rw-rw- 1 root root 21K Nov 26 13:18 all_pktproc_2_6.0.5_5.info

[gbogojevic]

Hi,

You have a couple of core dump files - all_pktproc_2 process on the dataplane crashed few times yesterday (this could be software bug). Now, it is running fine, but in order be sure and avoid some future outages, you should open a TAC case.

[MP18]

@avi_goldstain 

Also you can check the masterd daemon that is responsible for HA communications between 2 devices.

less mp-log masterd.log

Regards