snmpd crashing even after manual restart

Reply
Highlighted
L1 Bithead

snmpd crashing even after manual restart

Hi there,

 

We have 2 PA-3020 in our environment working as active/passive.
Recently after upgrading to PanOS9.04, both boxes are presenting issues with the snmp daemon. At first the problem was easily fixed with manual restart of process, and than became a weekly problem, but now the process last a few seconds before crashing.

 

[Last logs before snmp freezes]:
tail follow yes mp-log snmpd.log
2019-12-09 18:32:30 module:bfdMIB got id:14
2019-12-09 18:32:30 Current number of modules:14
2019-12-09 18:32:30 module:bfdMIB init'd
2019-12-09 18:32:30 Initializing module:panSSLBroker
2019-12-09 18:32:30 module:panSSLBroker got id:15
2019-12-09 18:32:30 Current number of modules:15
2019-12-09 18:32:30 sslbroker: there is zero broker stats entry
2019-12-09 18:32:30 sslbroker: there is zero broker status entry
2019-12-09 18:32:30 module:panSSLBroker init'd
2019-12-09 18:32:30 NET-SNMP version 5.7.2

 

[snmpd-debug]

2019-12-02 16:46:38.755 -0200 DEBUG: gdb: Calling crashInfoStart_event
2019-12-02 16:46:38.755 -0200 DEBUG: gdb: masterd_crash_info running with pid 30721
2019-12-02 16:46:38.755 -0200 DEBUG: gdb: adding timer crashInfoTimeout, event crashInfoTimeout in 300 seconds
2019-12-02 16:46:39.021 -0200 DEBUG: snmpd: Using crash info file /var/cores/crashinfo/snmpd_9.0.4_32.info
2019-12-02 16:46:44.535 -0200 DEBUG: gdb: Got event crash_info_finished_event
2019-12-02 16:46:44.537 -0200 DEBUG: gdb: Calling crash_info_finished_event
2019-12-02 16:46:44.537 -0200 DEBUG: gdb: Crash info finished for snmpd
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Got event crash_info_finished_event
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Calling crash_info_finished_event
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Crash info script finished
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: exiting from state running from user: False
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: transitioning from running -> exiting
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: exit time 1575312398.54, previous start time 1575312377.56 restart_reset 30
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: restarts 4 max 3 reboot-rule False, restart-en True
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: not restarting
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: transitioning from exiting -> stopped
2019-12-02 16:46:44.541 -0200 DEBUG: snmpd: sending childExited event to parent
2019-12-02 16:46:44.647 -0200 CRITICAL: snmpd: Exited 4 times, must be manually recovered.
2019-12-02 16:46:44.651 -0200 DEBUG: third_party: Got event childExited_event
2019-12-02 16:46:44.651 -0200 DEBUG: third_party: Calling childExited_event
2019-12-02 16:46:44.652 -0200 DEBUG: third_party: child snmpd exited our state running
2019-12-02 17:28:19.950 -0200 DEBUG: snmpd: Got event restart_event
2019-12-02 17:28:19.951 -0200 DEBUG: snmpd: Calling restart_event
2019-12-02 17:28:19.951 -0200 INFO: snmpd: received user restart
2019-12-02 17:28:19.952 -0200 INFO: snmpd: User restart reason - triggered by CLI
2019-12-02 17:28:19.952 -0200 INFO: snmpd: received user stop
2019-12-02 17:28:19.952 -0200 DEBUG: snmpd: transitioning from stopped -> stopping
2019-12-02 17:28:19.952 -0200 INFO: snmpd: exited, Core: True, Exit signal: SIGSEGV
2019-12-02 17:28:19.953 -0200 INFO: snmpd: logical exit-time 1575312398.54, actual exit-time 1575314899.95
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: looking for core for pid 30660
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: exiting from state stopping from user: True
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: transitioning from stopping -> exiting
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: exit time 1575312398.54, previous start time 1575312377.56 restart_reset 30
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: restarts 0 max 3 reboot-rule False, restart-en True
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: restarting
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no startScripts timer to remove
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no startScriptTimeout timer to remove
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no hbOn timer to remove
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no snmpdConf timer to remove
2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: transitioning from exiting -> restarting
2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: Adding process depends
2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: Exited 0 times, waiting 0 seconds before respawning.
2019-12-02 17:28:20.061 -0200 DEBUG: snmpd: adding timer deps, event start_helper in 0 seconds
2019-12-02 17:28:20.064 -0200 DEBUG: snmpd: Got event start_helper_event
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Got timer event deps, removing from active list
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Calling start_helper_event
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: transitioning from restarting -> scheduling
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Couldn't find cfg node md.apps.s0.mp.grp.websrvr.info
2019-12-02 17:28:20.067 -0200 DEBUG: snmpd: Got event sysdNotif_event

[mp-backtrace-snmpd]

less mp-backtrace snmpd_9.0.4_39.info

================================================================================
Process snmpd: Mon, 09 Dec 2019 17:41:16
================================================================================

-------------------------------------------------------------------------------
Backtrace
--------------------------------------------------------------------------------
[New LWP 27851]
[New LWP 27853]
[New LWP 27852]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/snmpd -I-hr_device hr_proc hr_network ifTable ifXTable ipAddressTable'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
[Current thread is 1 (Thread 0x7f7eaf421740 (LWP 27851))]

Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)):
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
#1 0x00007f7eaa9e3f84 in update_counter_data () at pan_globalCountersCache.c:509
#2 0x00007f7eaa9cb289 in handle_panGlobalCounters (handler=<optimized out>, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at panSys.c:2279
#3 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c280) at agent_handler.c:526
#4 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c370, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#5 0x00007f7eaefc00d9 in netsnmp_instance_helper_handler (handler=0x7f7eb0e3c370, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/instance.c:775
#6 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c370) at agent_handler.c:526
#7 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c3e0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#8 0x00007f7eaefc2af2 in netsnmp_scalar_helper_handler (handler=0x7f7eb0e3c3e0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/scalar.c:188
#9 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c3e0) at agent_handler.c:526
#10 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c450, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#11 0x00007f7eaefc32ed in netsnmp_serialize_helper_handler (handler=0x7f7eb0e3c450, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=<optimized out>) at helpers/serialize.c:66
#12 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c450) at agent_handler.c:526
#13 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c4c0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#14 0x00007f7eaefbd52a in netsnmp_bulk_to_next_helper (handler=0x7f7eb0e3c4c0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/bulk_to_next.c:110
#15 0x00007f7eaefd03ff in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c4c0) at agent_handler.c:526
#16 netsnmp_call_handlers (reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at agent_handler.c:611
#17 0x00007f7eaefddd79 in handle_var_requests (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:2689
#18 0x00007f7eaefde1c0 in handle_getnext_loop (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3148
#19 0x00007f7eaefdec30 in handle_pdu (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3477
#20 0x00007f7eaefded78 in netsnmp_handle_request (asp=asp@entry=0x7f7eb0ef0800, status=status@entry=0) at snmp_agent.c:3307
#21 0x00007f7eaefdf232 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x7f7eb0eb18b0, magic=<optimized out>) at snmp_agent.c:2000
#22 0x00007f7eae887ae1 in _sess_process_packet (sessp=sessp@entry=0x7f7eb0f0ccb0, sp=sp@entry=0x7f7eb0f0cce0, isp=isp@entry=0x7f7eb0f0c9b0, transport=transport@entry=0x7f7eb0f0ca20, opaque=<optimized out>, olength=<optimized out>, packetptr=packetptr@entry=0x7f7eb0f10220 "01\002\001\001\004\XXXXXXXXXXXXXX\245\037\002\004o\021t\324\002\001", length=51) at snmp_api.c:5437
#23 0x00007f7eae88866e in _sess_read (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5866
#24 0x00007f7eae8894b9 in snmp_sess_read2 (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5898
#25 0x00007f7eae88950b in snmp_read2 (fdset=0x7fffed6f5880) at snmp_api.c:5500
#26 0x00007f7eaf43c7b7 in receive () at snmpd.c:1361
#27 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1140

Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)):
#0 0x00007f7ead73b4e9 in __pselect (nfds=11, readfds=readfds@entry=0x7f7eb0d57b80, writefds=writefds@entry=0x7f7eb0d57c80, exceptfds=exceptfds@entry=0x7f7eb0d57d80, timeout=<optimized out>, timeout@entry=0x7f7ea73d2cf0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007f7ea98b9c1e in __evGetNext (opaqueCtx=opaqueCtx@entry=..., opaqueEv=opaqueEv@entry=0x7f7ea73d2ee0, options=options@entry=2) at eventlib.c:328
#2 0x00007f7ea98ba820 in __evMainLoop (opaqueCtx=...) at eventlib.c:654
#3 0x00007f7ea9a0f3fc in sysd_pthread_evloop (arg=0x7f7eb0d54460) at sysd_pthread.c:506
#4 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea73d3700) at pthread_create.c:308
#5 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f7ea9a1d80a in sysd_sched_get_work (sch=0x7f7eb0d63c00, tidx=tidx@entry=0) at sysd_sched.c:172
#2 0x00007f7ea9a0ee5f in sysd_pthread_worker (warg=0x7f7eb0d54460) at sysd_pthread.c:258
#3 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea77d4700) at pthread_create.c:308
#4 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

---SYSTEM ARCHITECTURE---
The target architecture is set automatically (currently i386:x86-64)

---SHARED MEMORY GDB---
From To Syms Read Shared Object Library
0x00007f7eaefbc740 0x00007f7eaeffc060 Yes /lib64/libnetsnmpagent.so.31
0x00007f7eaeb69e40 0x00007f7eaec235f4 Yes /lib64/libnetsnmpmibs.so.31
0x00007f7eae85fb80 0x00007f7eae8c972c Yes /lib64/libnetsnmp.so.31
0x00007f7eae642ed0 0x00007f7eae6439d0 Yes /lib64/libdl.so.2
0x00007f7eae3ee5f0 0x00007f7eae427438 Yes /lib64/librpm.so.3
0x00007f7eae1b9fa0 0x00007f7eae1cefe0 Yes /lib64/librpmio.so.3
0x00007f7eadd85900 0x00007f7eadf03af0 Yes /lib64/libcrypto.so.10
0x00007f7eada144b0 0x00007f7eada7e9e8 Yes /lib64/libm.so.6
0x00007f7ead66c3e0 0x00007f7ead7b0670 Yes /lib64/libc.so.6
0x00007f7eaf214ae0 0x00007f7eaf22f2ba Yes /lib64/ld-linux-x86-64.so.2
0x00007f7ead33f550 0x00007f7ead410a84 Yes (*) /lib64/libnss3.so
0x00007f7ead118760 0x00007f7ead1245f0 Yes (*) /lib64/libbz2.so.1
0x00007f7eacf03170 0x00007f7eacf0f6f0 Yes (*) /lib64/libz.so.1
0x00007f7eaccedb70 0x00007f7eaccfb9c8 Yes (*) /lib64/libelf.so.1
0x00007f7eacac8f30 0x00007f7eacadeea0 Yes (*) /lib64/liblzma.so.5
0x00007f7eac8bdcf0 0x00007f7eac8c32a8 Yes (*) /lib64/libpopt.so.0
0x00007f7eac677ca0 0x00007f7eac6a73f4 Yes /lib64/liblua.so
0x00007f7eac2e0cb0 0x00007f7eac420f5c Yes (*) /lib64/libdb-5.3.so
0x00007f7eac09b8a0 0x00007f7eac0a6514 Yes /lib64/libpthread.so.0
0x00007f7eabe75e30 0x00007f7eabe84318 Yes (*) /lib64/libnssutil3.so
0x00007f7eabc66510 0x00007f7eabc67b38 Yes (*) /lib64/libplc4.so
0x00007f7eaba62090 0x00007f7eaba63028 Yes (*) /lib64/libplds4.so
0x00007f7eab82fcd0 0x00007f7eab84fac0 Yes (*) /lib64/libnspr4.so
0x00007f7eab61d2c0 0x00007f7eab6200bc Yes /lib64/librt.so.1
0x00007f7eab40a240 0x00007f7eab411454 Yes /lib64/libnss_files.so.2
0x00007f7eab1ce2d0 0x00007f7eab1f79ac Yes (*) /lib64/libsoftokn3.so
0x00007f7eaaf1fa00 0x00007f7eaafa7860 Yes (*) /lib64/libsqlite3.so.0
0x00007f7eaac98900 0x00007f7eaaced9ad Yes (*) /lib64/libfreeblpriv3.so
0x00007f7eaa9c04d0 0x00007f7eaaa15080 Yes /usr/local/lib64/shobjs/libpanmib.so
0x00007f7eaa79f580 0x00007f7eaa79f668 Yes /lib64/libnetsnmphelpers.so.31
0x00007f7ea97aeb20 0x00007f7ea9a61d10 Yes /usr/local/lib64/libpancommon_mp.so.1
0x00007f7ea95148e0 0x00007f7ea95200fc Yes (*) /lib64/libexslt.so.0
0x00007f7ea92dcc60 0x00007f7ea93010d8 Yes (*) /lib64/libxslt.so.1
0x00007f7ea8f984f0 0x00007f7ea9080514 Yes (*) /lib64/libxml2.so.2
0x00007f7ea8d19870 0x00007f7ea8d520d4 Yes /lib64/libssl.so.10
0x00007f7ea8981120 0x00007f7ea8a60e20 Yes /lib64/libpython2.7.so.1.0
0x00007f7ea86cf9f0 0x00007f7ea8726a4c Yes (*) /lib64/libcurl.so.4
0x00007f7ea84655f0 0x00007f7ea84aa8b0 Yes (*) /lib64/libpcre.so.1
0x00007f7ea8260210 0x00007f7ea8261fe0 Yes /usr/local/lib64/3p/libcjson.so.1
0x00007f7ea805cf10 0x00007f7ea805d804 Yes /lib64/libutil.so.1
0x00007f7ea7e211c0 0x00007f7ea7e4b810 Yes /lib64/libldap-2.4.so.2
0x00007f7ea7c08610 0x00007f7ea7c0f444 Yes /lib64/liblber-2.4.so.2
0x00007f7ea79eea40 0x00007f7ea79fd854 Yes /lib64/libresolv.so.2
0x00007f7ea77d7af0 0x00007f7ea77e7298 Yes (*) /lib64/libgcc_s.so.1
(*): Shared library is missing debugging information.

---THREADS PC---

Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)):
=> 0x7f7ea9a22e2d <sysd_obj_release+29>: lock subl $0x1,0xc(%rdi)
0x7f7ea9a22e32 <sysd_obj_release+34>: je 0x7f7ea9a22e50 <sysd_obj_release+64>
0x7f7ea9a22e34 <sysd_obj_release+36>: mov 0x28(%rsp),%rax
0x7f7ea9a22e39 <sysd_obj_release+41>: xor %fs:0x28,%rax
0x7f7ea9a22e42 <sysd_obj_release+50>: jne 0x7f7ea9a2303c <sysd_obj_release+556>
0x7f7ea9a22e48 <sysd_obj_release+56>: add $0x30,%rsp
0x7f7ea9a22e4c <sysd_obj_release+60>: pop %rbx
0x7f7ea9a22e4d <sysd_obj_release+61>: retq

Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)):
=> 0x7f7ead73b4e9 <__pselect+169>: cmp $0xfffffffffffff000,%rax
0x7f7ead73b4ef <__pselect+175>: ja 0x7f7ead73b517 <__pselect+215>
0x7f7ead73b4f1 <__pselect+177>: mov %ebp,%edi
0x7f7ead73b4f3 <__pselect+179>: mov %eax,0x8(%rsp)
0x7f7ead73b4f7 <__pselect+183>: callq 0x7f7ead751740 <__libc_disable_asynccancel>
0x7f7ead73b4fc <__pselect+188>: mov 0x8(%rsp),%eax
0x7f7ead73b500 <__pselect+192>: jmp 0x7f7ead73b493 <__pselect+83>
0x7f7ead73b502 <__pselect+194>: mov 0x2cc95f(%rip),%rdx # 0x7f7eada07e68

Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)):
=> 0x7f7eac0a16d5 <pthread_cond_wait@@GLIBC_2.3.2+197>: mov (%rsp),%edi
0x7f7eac0a16d8 <pthread_cond_wait@@GLIBC_2.3.2+200>: callq 0x7f7eac0a3ea0 <__pthread_disable_asynccancel>
0x7f7eac0a16dd <pthread_cond_wait@@GLIBC_2.3.2+205>: mov 0x8(%rsp),%rdi
0x7f7eac0a16e2 <pthread_cond_wait@@GLIBC_2.3.2+210>: mov $0x1,%esi
0x7f7eac0a16e7 <pthread_cond_wait@@GLIBC_2.3.2+215>: xor %eax,%eax
0x7f7eac0a16e9 <pthread_cond_wait@@GLIBC_2.3.2+217>: lock cmpxchg %esi,(%rdi)
0x7f7eac0a16ed <pthread_cond_wait@@GLIBC_2.3.2+221>: jne 0x7f7eac0a17d3 <pthread_cond_wait@@GLIBC_2.3.2+451>
0x7f7eac0a16f3 <pthread_cond_wait@@GLIBC_2.3.2+227>: mov 0x2c(%rdi),%edx

---THREADS SP---

cfg.platform.portcount: 20
cfg.platform.serial:
cfg.platform.total-slots: 1
cfg.platform.variant: base
cfg.platform.version: 1.1
cfg.platform.vpn-disable: False
cfg.net.s0.eth0.cfg:
4, }, 'up': True, 'v6routes': { 'default': , }, }
ha.app.local.state: passive
ha.app.peer.state: active

--------------------------------------------------------------------------------
Backtrace
--------------------------------------------------------------------------------
[New LWP 27851]
[New LWP 27853]
[New LWP 27852]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/snmpd -I-hr_device hr_proc hr_network ifTable ifXTable ipAddressTable'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
[Current thread is 1 (Thread 0x7f7eaf421740 (LWP 27851))]

Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)):
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
#1 0x00007f7eaa9e3f84 in update_counter_data () at pan_globalCountersCache.c:509
#2 0x00007f7eaa9cb289 in handle_panGlobalCounters (handler=<optimized out>, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at panSys.c:2279
#3 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c280) at agent_handler.c:526
#4 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c370, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#5 0x00007f7eaefc00d9 in netsnmp_instance_helper_handler (handler=0x7f7eb0e3c370, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/instance.c:775
#6 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c370) at agent_handler.c:526
#7 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c3e0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#8 0x00007f7eaefc2af2 in netsnmp_scalar_helper_handler (handler=0x7f7eb0e3c3e0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/scalar.c:188
#9 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c3e0) at agent_handler.c:526
#10 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c450, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#11 0x00007f7eaefc32ed in netsnmp_serialize_helper_handler (handler=0x7f7eb0e3c450, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=<optimized out>) at helpers/serialize.c:66
#12 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c450) at agent_handler.c:526
#13 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c4c0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#14 0x00007f7eaefbd52a in netsnmp_bulk_to_next_helper (handler=0x7f7eb0e3c4c0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/bulk_to_next.c:110
#15 0x00007f7eaefd03ff in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c4c0) at agent_handler.c:526
#16 netsnmp_call_handlers (reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at agent_handler.c:611
#17 0x00007f7eaefddd79 in handle_var_requests (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:2689
#18 0x00007f7eaefde1c0 in handle_getnext_loop (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3148
#19 0x00007f7eaefdec30 in handle_pdu (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3477
#20 0x00007f7eaefded78 in netsnmp_handle_request (asp=asp@entry=0x7f7eb0ef0800, status=status@entry=0) at snmp_agent.c:3307
#21 0x00007f7eaefdf232 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x7f7eb0eb18b0, magic=<optimized out>) at snmp_agent.c:2000
#22 0x00007f7eae887ae1 in _sess_process_packet (sessp=sessp@entry=0x7f7eb0f0ccb0, sp=sp@entry=0x7f7eb0f0cce0, isp=isp@entry=0x7f7eb0f0c9b0, transport=transport@entry=0x7f7eb0f0ca20, opaque=<optimized out>, oleng
th=<optimized out>, packetptr=packetptr@entry=0x7f7eb0f10220 "01\002\001\001\004\XXXXXXXXXXXXXXXXXXX\245\037\002\004o\021t\324\002\001", length=51) at snmp_api.c:5437
#23 0x00007f7eae88866e in _sess_read (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5866
#24 0x00007f7eae8894b9 in snmp_sess_read2 (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5898
#25 0x00007f7eae88950b in snmp_read2 (fdset=0x7fffed6f5880) at snmp_api.c:5500
#26 0x00007f7eaf43c7b7 in receive () at snmpd.c:1361
#27 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1140

Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)):
#0 0x00007f7ead73b4e9 in __pselect (nfds=11, readfds=readfds@entry=0x7f7eb0d57b80, writefds=writefds@entry=0x7f7eb0d57c80, exceptfds=exceptfds@entry=0x7f7eb0d57d80, timeout=<optimized out>, timeout@entry=0x7f7ea73d2cf0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007f7ea98b9c1e in __evGetNext (opaqueCtx=opaqueCtx@entry=..., opaqueEv=opaqueEv@entry=0x7f7ea73d2ee0, options=options@entry=2) at eventlib.c:328
#2 0x00007f7ea98ba820 in __evMainLoop (opaqueCtx=...) at eventlib.c:654
#3 0x00007f7ea9a0f3fc in sysd_pthread_evloop (arg=0x7f7eb0d54460) at sysd_pthread.c:506
#4 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea73d3700) at pthread_create.c:308
#5 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f7ea9a1d80a in sysd_sched_get_work (sch=0x7f7eb0d63c00, tidx=tidx@entry=0) at sysd_sched.c:172
#2 0x00007f7ea9a0ee5f in sysd_pthread_worker (warg=0x7f7eb0d54460) at sysd_pthread.c:258
#3 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea77d4700) at pthread_create.c:308
XXXXXXXXXXXX(passive)> tail follow yes mp-log masterd_detail.log
2019-12-09 18:10:02.639 -0200 DEBUG: stats_gen: exiting from state running from user: False
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: transitioning from running -> exiting
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: exit time 1575922202.54, previous start time 1575922201.35 restart_reset 300
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: restarts 0 max 3 reboot-rule False, restart-en True
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: not restarting
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: transitioning from exiting -> notRunning
2019-12-09 18:10:02.641 -0200 DEBUG: stats_gen: sending childExited event to parent
2019-12-09 18:10:02.644 -0200 DEBUG: batch: Got event childExited_event
2019-12-09 18:10:02.644 -0200 DEBUG: batch: Calling childExited_event
2019-12-09 18:10:02.645 -0200 DEBUG: batch: child stats_gen exited pid 18082

 

Thanks in advance

Highlighted
Community Team Member

Re: snmpd crashing even after manual restart

Hi @UNIRIO ,

 

Please have the corefile analysed by support :

 

How to export core files. 

or

How to upload core files to Support directly 

 

Cheers !

-Kiwi.

Highlighted
L1 Bithead

Re: snmpd crashing even after manual restart

Hi @kiwi ,

unfortunately we can't upload the file nor open a case with Palo Alto because of Premium Support we must contact our Authorized Support Center (ASC).

 

Note: Have the case number and the email of the case contact nearby--these items are required to authenticate the user to the tacupload server.

 

Thanks!

Highlighted
Community Team Member

Re: snmpd crashing even after manual restart

Hi @UNIRIO ,

 

That's unfortunate.  Eitherway the corefile needs to be analysed to be conclusive.

If possible, provide the TSF, backtrace and corefile to your ASC and have them open a support case with TAC on your behalf.

 

Cheers !

-Kiwi,

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!