snmpd crashing even after manual restart

[UNIRIO]

Hi there,

We have 2 PA-3020 in our environment working as active/passive.
Recently after upgrading to PanOS9.04, both boxes are presenting issues with the snmp daemon. At first the problem was easily fixed with manual restart of process, and than became a weekly problem, but now the process last a few seconds before crashing.

[Last logs before snmp freezes]:
tail follow yes mp-log snmpd.log
2019-12-09 18:32:30 module:bfdMIB got id:14
2019-12-09 18:32:30 Current number of modules:14
2019-12-09 18:32:30 module:bfdMIB init'd
2019-12-09 18:32:30 Initializing module:panSSLBroker
2019-12-09 18:32:30 module:panSSLBroker got id:15
2019-12-09 18:32:30 Current number of modules:15
2019-12-09 18:32:30 sslbroker: there is zero broker stats entry
2019-12-09 18:32:30 sslbroker: there is zero broker status entry
2019-12-09 18:32:30 module:panSSLBroker init'd
2019-12-09 18:32:30 NET-SNMP version 5.7.2

[snmpd-debug]

2019-12-02 16:46:38.755 -0200 DEBUG: gdb: Calling crashInfoStart_event
2019-12-02 16:46:38.755 -0200 DEBUG: gdb: masterd_crash_info running with pid 30721
2019-12-02 16:46:38.755 -0200 DEBUG: gdb: adding timer crashInfoTimeout, event crashInfoTimeout in 300 seconds
2019-12-02 16:46:39.021 -0200 DEBUG: snmpd: Using crash info file /var/cores/crashinfo/snmpd_9.0.4_32.info
2019-12-02 16:46:44.535 -0200 DEBUG: gdb: Got event crash_info_finished_event
2019-12-02 16:46:44.537 -0200 DEBUG: gdb: Calling crash_info_finished_event
2019-12-02 16:46:44.537 -0200 DEBUG: gdb: Crash info finished for snmpd
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Got event crash_info_finished_event
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Calling crash_info_finished_event
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Crash info script finished
2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: exiting from state running from user: False
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: transitioning from running -> exiting
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: exit time 1575312398.54, previous start time 1575312377.56 restart_reset 30
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: restarts 4 max 3 reboot-rule False, restart-en True
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: not restarting
2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: transitioning from exiting -> stopped
2019-12-02 16:46:44.541 -0200 DEBUG: snmpd: sending childExited event to parent
2019-12-02 16:46:44.647 -0200 CRITICAL: snmpd: Exited 4 times, must be manually recovered.
2019-12-02 16:46:44.651 -0200 DEBUG: third_party: Got event childExited_event
2019-12-02 16:46:44.651 -0200 DEBUG: third_party: Calling childExited_event
2019-12-02 16:46:44.652 -0200 DEBUG: third_party: child snmpd exited our state running
2019-12-02 17:28:19.950 -0200 DEBUG: snmpd: Got event restart_event
2019-12-02 17:28:19.951 -0200 DEBUG: snmpd: Calling restart_event
2019-12-02 17:28:19.951 -0200 INFO: snmpd: received user restart
2019-12-02 17:28:19.952 -0200 INFO: snmpd: User restart reason - triggered by CLI
2019-12-02 17:28:19.952 -0200 INFO: snmpd: received user stop
2019-12-02 17:28:19.952 -0200 DEBUG: snmpd: transitioning from stopped -> stopping
2019-12-02 17:28:19.952 -0200 INFO: snmpd: exited, Core: True, Exit signal: SIGSEGV
2019-12-02 17:28:19.953 -0200 INFO: snmpd: logical exit-time 1575312398.54, actual exit-time 1575314899.95
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: looking for core for pid 30660
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: exiting from state stopping from user: True
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: transitioning from stopping -> exiting
2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: exit time 1575312398.54, previous start time 1575312377.56 restart_reset 30
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: restarts 0 max 3 reboot-rule False, restart-en True
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: restarting
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no startScripts timer to remove
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no startScriptTimeout timer to remove
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no hbOn timer to remove
2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no snmpdConf timer to remove
2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: transitioning from exiting -> restarting
2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: Adding process depends
2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: Exited 0 times, waiting 0 seconds before respawning.
2019-12-02 17:28:20.061 -0200 DEBUG: snmpd: adding timer deps, event start_helper in 0 seconds
2019-12-02 17:28:20.064 -0200 DEBUG: snmpd: Got event start_helper_event
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Got timer event deps, removing from active list
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Calling start_helper_event
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: transitioning from restarting -> scheduling
2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Couldn't find cfg node md.apps.s0.mp.grp.websrvr.info
2019-12-02 17:28:20.067 -0200 DEBUG: snmpd: Got event sysdNotif_event

[mp-backtrace-snmpd]

less mp-backtrace snmpd_9.0.4_39.info

================================================================================
Process snmpd: Mon, 09 Dec 2019 17:41:16
================================================================================

-------------------------------------------------------------------------------
Backtrace
--------------------------------------------------------------------------------
[New LWP 27851]
[New LWP 27853]
[New LWP 27852]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/snmpd -I-hr_device hr_proc hr_network ifTable ifXTable ipAddressTable'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
[Current thread is 1 (Thread 0x7f7eaf421740 (LWP 27851))]

Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)):
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
#1 0x00007f7eaa9e3f84 in update_counter_data () at pan_globalCountersCache.c:509
#2 0x00007f7eaa9cb289 in handle_panGlobalCounters (handler=<optimized out>, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at panSys.c:2279
#3 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c280) at agent_handler.c:526
#4 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c370, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#5 0x00007f7eaefc00d9 in netsnmp_instance_helper_handler (handler=0x7f7eb0e3c370, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/instance.c:775
#6 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c370) at agent_handler.c:526
#7 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c3e0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#8 0x00007f7eaefc2af2 in netsnmp_scalar_helper_handler (handler=0x7f7eb0e3c3e0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/scalar.c:188
#9 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c3e0) at agent_handler.c:526
#10 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c450, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#11 0x00007f7eaefc32ed in netsnmp_serialize_helper_handler (handler=0x7f7eb0e3c450, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=<optimized out>) at helpers/serialize.c:66
#12 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c450) at agent_handler.c:526
#13 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c4c0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#14 0x00007f7eaefbd52a in netsnmp_bulk_to_next_helper (handler=0x7f7eb0e3c4c0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/bulk_to_next.c:110
#15 0x00007f7eaefd03ff in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c4c0) at agent_handler.c:526
#16 netsnmp_call_handlers (reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at agent_handler.c:611
#17 0x00007f7eaefddd79 in handle_var_requests (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:2689
#18 0x00007f7eaefde1c0 in handle_getnext_loop (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3148
#19 0x00007f7eaefdec30 in handle_pdu (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3477
#20 0x00007f7eaefded78 in netsnmp_handle_request (asp=asp@entry=0x7f7eb0ef0800, status=status@entry=0) at snmp_agent.c:3307
#21 0x00007f7eaefdf232 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x7f7eb0eb18b0, magic=<optimized out>) at snmp_agent.c:2000
#22 0x00007f7eae887ae1 in _sess_process_packet (sessp=sessp@entry=0x7f7eb0f0ccb0, sp=sp@entry=0x7f7eb0f0cce0, isp=isp@entry=0x7f7eb0f0c9b0, transport=transport@entry=0x7f7eb0f0ca20, opaque=<optimized out>, olength=<optimized out>, packetptr=packetptr@entry=0x7f7eb0f10220 "01\002\001\001\004\XXXXXXXXXXXXXX\245\037\002\004o\021t\324\002\001", length=51) at snmp_api.c:5437
#23 0x00007f7eae88866e in _sess_read (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5866
#24 0x00007f7eae8894b9 in snmp_sess_read2 (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5898
#25 0x00007f7eae88950b in snmp_read2 (fdset=0x7fffed6f5880) at snmp_api.c:5500
#26 0x00007f7eaf43c7b7 in receive () at snmpd.c:1361
#27 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1140

Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)):
#0 0x00007f7ead73b4e9 in __pselect (nfds=11, readfds=readfds@entry=0x7f7eb0d57b80, writefds=writefds@entry=0x7f7eb0d57c80, exceptfds=exceptfds@entry=0x7f7eb0d57d80, timeout=<optimized out>, timeout@entry=0x7f7ea73d2cf0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007f7ea98b9c1e in __evGetNext (opaqueCtx=opaqueCtx@entry=..., opaqueEv=opaqueEv@entry=0x7f7ea73d2ee0, options=options@entry=2) at eventlib.c:328
#2 0x00007f7ea98ba820 in __evMainLoop (opaqueCtx=...) at eventlib.c:654
#3 0x00007f7ea9a0f3fc in sysd_pthread_evloop (arg=0x7f7eb0d54460) at sysd_pthread.c:506
#4 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea73d3700) at pthread_create.c:308
#5 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f7ea9a1d80a in sysd_sched_get_work (sch=0x7f7eb0d63c00, tidx=tidx@entry=0) at sysd_sched.c:172
#2 0x00007f7ea9a0ee5f in sysd_pthread_worker (warg=0x7f7eb0d54460) at sysd_pthread.c:258
#3 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea77d4700) at pthread_create.c:308
#4 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

---SYSTEM ARCHITECTURE---
The target architecture is set automatically (currently i386:x86-64)

---SHARED MEMORY GDB---
From To Syms Read Shared Object Library
0x00007f7eaefbc740 0x00007f7eaeffc060 Yes /lib64/libnetsnmpagent.so.31
0x00007f7eaeb69e40 0x00007f7eaec235f4 Yes /lib64/libnetsnmpmibs.so.31
0x00007f7eae85fb80 0x00007f7eae8c972c Yes /lib64/libnetsnmp.so.31
0x00007f7eae642ed0 0x00007f7eae6439d0 Yes /lib64/libdl.so.2
0x00007f7eae3ee5f0 0x00007f7eae427438 Yes /lib64/librpm.so.3
0x00007f7eae1b9fa0 0x00007f7eae1cefe0 Yes /lib64/librpmio.so.3
0x00007f7eadd85900 0x00007f7eadf03af0 Yes /lib64/libcrypto.so.10
0x00007f7eada144b0 0x00007f7eada7e9e8 Yes /lib64/libm.so.6
0x00007f7ead66c3e0 0x00007f7ead7b0670 Yes /lib64/libc.so.6
0x00007f7eaf214ae0 0x00007f7eaf22f2ba Yes /lib64/ld-linux-x86-64.so.2
0x00007f7ead33f550 0x00007f7ead410a84 Yes (*) /lib64/libnss3.so
0x00007f7ead118760 0x00007f7ead1245f0 Yes (*) /lib64/libbz2.so.1
0x00007f7eacf03170 0x00007f7eacf0f6f0 Yes (*) /lib64/libz.so.1
0x00007f7eaccedb70 0x00007f7eaccfb9c8 Yes (*) /lib64/libelf.so.1
0x00007f7eacac8f30 0x00007f7eacadeea0 Yes (*) /lib64/liblzma.so.5
0x00007f7eac8bdcf0 0x00007f7eac8c32a8 Yes (*) /lib64/libpopt.so.0
0x00007f7eac677ca0 0x00007f7eac6a73f4 Yes /lib64/liblua.so
0x00007f7eac2e0cb0 0x00007f7eac420f5c Yes (*) /lib64/libdb-5.3.so
0x00007f7eac09b8a0 0x00007f7eac0a6514 Yes /lib64/libpthread.so.0
0x00007f7eabe75e30 0x00007f7eabe84318 Yes (*) /lib64/libnssutil3.so
0x00007f7eabc66510 0x00007f7eabc67b38 Yes (*) /lib64/libplc4.so
0x00007f7eaba62090 0x00007f7eaba63028 Yes (*) /lib64/libplds4.so
0x00007f7eab82fcd0 0x00007f7eab84fac0 Yes (*) /lib64/libnspr4.so
0x00007f7eab61d2c0 0x00007f7eab6200bc Yes /lib64/librt.so.1
0x00007f7eab40a240 0x00007f7eab411454 Yes /lib64/libnss_files.so.2
0x00007f7eab1ce2d0 0x00007f7eab1f79ac Yes (*) /lib64/libsoftokn3.so
0x00007f7eaaf1fa00 0x00007f7eaafa7860 Yes (*) /lib64/libsqlite3.so.0
0x00007f7eaac98900 0x00007f7eaaced9ad Yes (*) /lib64/libfreeblpriv3.so
0x00007f7eaa9c04d0 0x00007f7eaaa15080 Yes /usr/local/lib64/shobjs/libpanmib.so
0x00007f7eaa79f580 0x00007f7eaa79f668 Yes /lib64/libnetsnmphelpers.so.31
0x00007f7ea97aeb20 0x00007f7ea9a61d10 Yes /usr/local/lib64/libpancommon_mp.so.1
0x00007f7ea95148e0 0x00007f7ea95200fc Yes (*) /lib64/libexslt.so.0
0x00007f7ea92dcc60 0x00007f7ea93010d8 Yes (*) /lib64/libxslt.so.1
0x00007f7ea8f984f0 0x00007f7ea9080514 Yes (*) /lib64/libxml2.so.2
0x00007f7ea8d19870 0x00007f7ea8d520d4 Yes /lib64/libssl.so.10
0x00007f7ea8981120 0x00007f7ea8a60e20 Yes /lib64/libpython2.7.so.1.0
0x00007f7ea86cf9f0 0x00007f7ea8726a4c Yes (*) /lib64/libcurl.so.4
0x00007f7ea84655f0 0x00007f7ea84aa8b0 Yes (*) /lib64/libpcre.so.1
0x00007f7ea8260210 0x00007f7ea8261fe0 Yes /usr/local/lib64/3p/libcjson.so.1
0x00007f7ea805cf10 0x00007f7ea805d804 Yes /lib64/libutil.so.1
0x00007f7ea7e211c0 0x00007f7ea7e4b810 Yes /lib64/libldap-2.4.so.2
0x00007f7ea7c08610 0x00007f7ea7c0f444 Yes /lib64/liblber-2.4.so.2
0x00007f7ea79eea40 0x00007f7ea79fd854 Yes /lib64/libresolv.so.2
0x00007f7ea77d7af0 0x00007f7ea77e7298 Yes (*) /lib64/libgcc_s.so.1
(*): Shared library is missing debugging information.

---THREADS PC---

Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)):
=> 0x7f7ea9a22e2d <sysd_obj_release+29>: lock subl $0x1,0xc(%rdi)
0x7f7ea9a22e32 <sysd_obj_release+34>: je 0x7f7ea9a22e50 <sysd_obj_release+64>
0x7f7ea9a22e34 <sysd_obj_release+36>: mov 0x28(%rsp),%rax
0x7f7ea9a22e39 <sysd_obj_release+41>: xor %fs:0x28,%rax
0x7f7ea9a22e42 <sysd_obj_release+50>: jne 0x7f7ea9a2303c <sysd_obj_release+556>
0x7f7ea9a22e48 <sysd_obj_release+56>: add $0x30,%rsp
0x7f7ea9a22e4c <sysd_obj_release+60>: pop %rbx
0x7f7ea9a22e4d <sysd_obj_release+61>: retq

Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)):
=> 0x7f7ead73b4e9 <__pselect+169>: cmp $0xfffffffffffff000,%rax
0x7f7ead73b4ef <__pselect+175>: ja 0x7f7ead73b517 <__pselect+215>
0x7f7ead73b4f1 <__pselect+177>: mov %ebp,%edi
0x7f7ead73b4f3 <__pselect+179>: mov %eax,0x8(%rsp)
0x7f7ead73b4f7 <__pselect+183>: callq 0x7f7ead751740 <__libc_disable_asynccancel>
0x7f7ead73b4fc <__pselect+188>: mov 0x8(%rsp),%eax
0x7f7ead73b500 <__pselect+192>: jmp 0x7f7ead73b493 <__pselect+83>
0x7f7ead73b502 <__pselect+194>: mov 0x2cc95f(%rip),%rdx # 0x7f7eada07e68

Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)):
=> 0x7f7eac0a16d5 <pthread_cond_wait@@GLIBC_2.3.2+197>: mov (%rsp),%edi
0x7f7eac0a16d8 <pthread_cond_wait@@GLIBC_2.3.2+200>: callq 0x7f7eac0a3ea0 <__pthread_disable_asynccancel>
0x7f7eac0a16dd <pthread_cond_wait@@GLIBC_2.3.2+205>: mov 0x8(%rsp),%rdi
0x7f7eac0a16e2 <pthread_cond_wait@@GLIBC_2.3.2+210>: mov $0x1,%esi
0x7f7eac0a16e7 <pthread_cond_wait@@GLIBC_2.3.2+215>: xor %eax,%eax
0x7f7eac0a16e9 <pthread_cond_wait@@GLIBC_2.3.2+217>: lock cmpxchg %esi,(%rdi)
0x7f7eac0a16ed <pthread_cond_wait@@GLIBC_2.3.2+221>: jne 0x7f7eac0a17d3 <pthread_cond_wait@@GLIBC_2.3.2+451>
0x7f7eac0a16f3 <pthread_cond_wait@@GLIBC_2.3.2+227>: mov 0x2c(%rdi),%edx

---THREADS SP---

cfg.platform.portcount: 20
cfg.platform.serial:
cfg.platform.total-slots: 1
cfg.platform.variant: base
cfg.platform.version: 1.1
cfg.platform.vpn-disable: False
cfg.net.s0.eth0.cfg:
4, }, 'up': True, 'v6routes': { 'default': , }, }
ha.app.local.state: passive
ha.app.peer.state: active

--------------------------------------------------------------------------------
Backtrace
--------------------------------------------------------------------------------
[New LWP 27851]
[New LWP 27853]
[New LWP 27852]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/snmpd -I-hr_device hr_proc hr_network ifTable ifXTable ipAddressTable'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
[Current thread is 1 (Thread 0x7f7eaf421740 (LWP 27851))]

Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)):
#0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279
#1 0x00007f7eaa9e3f84 in update_counter_data () at pan_globalCountersCache.c:509
#2 0x00007f7eaa9cb289 in handle_panGlobalCounters (handler=<optimized out>, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at panSys.c:2279
#3 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c280) at agent_handler.c:526
#4 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c370, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#5 0x00007f7eaefc00d9 in netsnmp_instance_helper_handler (handler=0x7f7eb0e3c370, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/instance.c:775
#6 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c370) at agent_handler.c:526
#7 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c3e0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#8 0x00007f7eaefc2af2 in netsnmp_scalar_helper_handler (handler=0x7f7eb0e3c3e0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/scalar.c:188
#9 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c3e0) at agent_handler.c:526
#10 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c450, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#11 0x00007f7eaefc32ed in netsnmp_serialize_helper_handler (handler=0x7f7eb0e3c450, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=<optimized out>) at helpers/serialize.c:66
#12 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c450) at agent_handler.c:526
#13 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c4c0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640
#14 0x00007f7eaefbd52a in netsnmp_bulk_to_next_helper (handler=0x7f7eb0e3c4c0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/bulk_to_next.c:110
#15 0x00007f7eaefd03ff in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c4c0) at agent_handler.c:526
#16 netsnmp_call_handlers (reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at agent_handler.c:611
#17 0x00007f7eaefddd79 in handle_var_requests (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:2689
#18 0x00007f7eaefde1c0 in handle_getnext_loop (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3148
#19 0x00007f7eaefdec30 in handle_pdu (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3477
#20 0x00007f7eaefded78 in netsnmp_handle_request (asp=asp@entry=0x7f7eb0ef0800, status=status@entry=0) at snmp_agent.c:3307
#21 0x00007f7eaefdf232 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x7f7eb0eb18b0, magic=<optimized out>) at snmp_agent.c:2000
#22 0x00007f7eae887ae1 in _sess_process_packet (sessp=sessp@entry=0x7f7eb0f0ccb0, sp=sp@entry=0x7f7eb0f0cce0, isp=isp@entry=0x7f7eb0f0c9b0, transport=transport@entry=0x7f7eb0f0ca20, opaque=<optimized out>, oleng
th=<optimized out>, packetptr=packetptr@entry=0x7f7eb0f10220 "01\002\001\001\004\XXXXXXXXXXXXXXXXXXX\245\037\002\004o\021t\324\002\001", length=51) at snmp_api.c:5437
#23 0x00007f7eae88866e in _sess_read (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5866
#24 0x00007f7eae8894b9 in snmp_sess_read2 (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5898
#25 0x00007f7eae88950b in snmp_read2 (fdset=0x7fffed6f5880) at snmp_api.c:5500
#26 0x00007f7eaf43c7b7 in receive () at snmpd.c:1361
#27 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1140

Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)):
#0 0x00007f7ead73b4e9 in __pselect (nfds=11, readfds=readfds@entry=0x7f7eb0d57b80, writefds=writefds@entry=0x7f7eb0d57c80, exceptfds=exceptfds@entry=0x7f7eb0d57d80, timeout=<optimized out>, timeout@entry=0x7f7ea73d2cf0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007f7ea98b9c1e in __evGetNext (opaqueCtx=opaqueCtx@entry=..., opaqueEv=opaqueEv@entry=0x7f7ea73d2ee0, options=options@entry=2) at eventlib.c:328
#2 0x00007f7ea98ba820 in __evMainLoop (opaqueCtx=...) at eventlib.c:654
#3 0x00007f7ea9a0f3fc in sysd_pthread_evloop (arg=0x7f7eb0d54460) at sysd_pthread.c:506
#4 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea73d3700) at pthread_create.c:308
#5 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f7ea9a1d80a in sysd_sched_get_work (sch=0x7f7eb0d63c00, tidx=tidx@entry=0) at sysd_sched.c:172
#2 0x00007f7ea9a0ee5f in sysd_pthread_worker (warg=0x7f7eb0d54460) at sysd_pthread.c:258
#3 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea77d4700) at pthread_create.c:308
XXXXXXXXXXXX(passive)> tail follow yes mp-log masterd_detail.log
2019-12-09 18:10:02.639 -0200 DEBUG: stats_gen: exiting from state running from user: False
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: transitioning from running -> exiting
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: exit time 1575922202.54, previous start time 1575922201.35 restart_reset 300
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: restarts 0 max 3 reboot-rule False, restart-en True
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: not restarting
2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: transitioning from exiting -> notRunning
2019-12-09 18:10:02.641 -0200 DEBUG: stats_gen: sending childExited event to parent
2019-12-09 18:10:02.644 -0200 DEBUG: batch: Got event childExited_event
2019-12-09 18:10:02.644 -0200 DEBUG: batch: Calling childExited_event
2019-12-09 18:10:02.645 -0200 DEBUG: batch: child stats_gen exited pid 18082

Thanks in advance

[kiwi]

Hi @UNIRIO ,

Please have the corefile analysed by support :

How to export core files. 

or

How to upload core files to Support directly 

Cheers !

-Kiwi.

[UNIRIO]

Hi @kiwi ,

unfortunately we can't upload the file nor open a case with Palo Alto because of Premium Support we must contact our Authorized Support Center (ASC).

Note: Have the case number and the email of the case contact nearby--these items are required to authenticate the user to the tacupload server.

Thanks!

[kiwi]

Hi @UNIRIO ,

That's unfortunate.  Eitherway the corefile needs to be analysed to be conclusive.

If possible, provide the TSF, backtrace and corefile to your ASC and have them open a support case with TAC on your behalf.

Cheers !

-Kiwi,