About UNIRIO

Hi there,   We have 2 PA-3020 in our environment working as active/passive. Recently after upgrading to PanOS9.04, both boxes are presenting issues with the snmp daemon. At first the problem was easily fixed with manual restart of process, and than became a weekly problem, but now the process last a few seconds before crashing.   [Last logs before snmp freezes]: tail follow yes mp-log snmpd.log 2019-12-09 18:32:30 module:bfdMIB got id:14 2019-12-09 18:32:30 Current number of modules:14 2019-12-09 18:32:30 module:bfdMIB init'd 2019-12-09 18:32:30 Initializing module:panSSLBroker 2019-12-09 18:32:30 module:panSSLBroker got id:15 2019-12-09 18:32:30 Current number of modules:15 2019-12-09 18:32:30 sslbroker: there is zero broker stats entry 2019-12-09 18:32:30 sslbroker: there is zero broker status entry 2019-12-09 18:32:30 module:panSSLBroker init'd 2019-12-09 18:32:30 NET-SNMP version 5.7.2   [snmpd-debug] 2019-12-02 16:46:38.755 -0200 DEBUG: gdb: Calling crashInfoStart_event 2019-12-02 16:46:38.755 -0200 DEBUG: gdb: masterd_crash_info running with pid 30721 2019-12-02 16:46:38.755 -0200 DEBUG: gdb: adding timer crashInfoTimeout, event crashInfoTimeout in 300 seconds 2019-12-02 16:46:39.021 -0200 DEBUG: snmpd: Using crash info file /var/cores/crashinfo/snmpd_9.0.4_32.info 2019-12-02 16:46:44.535 -0200 DEBUG: gdb: Got event crash_info_finished_event 2019-12-02 16:46:44.537 -0200 DEBUG: gdb: Calling crash_info_finished_event 2019-12-02 16:46:44.537 -0200 DEBUG: gdb: Crash info finished for snmpd 2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Got event crash_info_finished_event 2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Calling crash_info_finished_event 2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: Crash info script finished 2019-12-02 16:46:44.539 -0200 DEBUG: snmpd: exiting from state running from user: False 2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: transitioning from running -> exiting 2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: exit time 1575312398.54, previous start time 1575312377.56 restart_reset 30 2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: restarts 4 max 3 reboot-rule False, restart-en True 2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: not restarting 2019-12-02 16:46:44.540 -0200 DEBUG: snmpd: transitioning from exiting -> stopped 2019-12-02 16:46:44.541 -0200 DEBUG: snmpd: sending childExited event to parent 2019-12-02 16:46:44.647 -0200 CRITICAL: snmpd: Exited 4 times, must be manually recovered. 2019-12-02 16:46:44.651 -0200 DEBUG: third_party: Got event childExited_event 2019-12-02 16:46:44.651 -0200 DEBUG: third_party: Calling childExited_event 2019-12-02 16:46:44.652 -0200 DEBUG: third_party: child snmpd exited our state running 2019-12-02 17:28:19.950 -0200 DEBUG: snmpd: Got event restart_event 2019-12-02 17:28:19.951 -0200 DEBUG: snmpd: Calling restart_event 2019-12-02 17:28:19.951 -0200 INFO: snmpd: received user restart 2019-12-02 17:28:19.952 -0200 INFO: snmpd: User restart reason - triggered by CLI 2019-12-02 17:28:19.952 -0200 INFO: snmpd: received user stop 2019-12-02 17:28:19.952 -0200 DEBUG: snmpd: transitioning from stopped -> stopping 2019-12-02 17:28:19.952 -0200 INFO: snmpd: exited, Core: True, Exit signal: SIGSEGV 2019-12-02 17:28:19.953 -0200 INFO: snmpd: logical exit-time 1575312398.54, actual exit-time 1575314899.95 2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: looking for core for pid 30660 2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: exiting from state stopping from user: True 2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: transitioning from stopping -> exiting 2019-12-02 17:28:20.058 -0200 DEBUG: snmpd: exit time 1575312398.54, previous start time 1575312377.56 restart_reset 30 2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: restarts 0 max 3 reboot-rule False, restart-en True 2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: restarting 2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no startScripts timer to remove 2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no startScriptTimeout timer to remove 2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no hbOn timer to remove 2019-12-02 17:28:20.059 -0200 DEBUG: snmpd: stopping, no snmpdConf timer to remove 2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: transitioning from exiting -> restarting 2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: Adding process depends 2019-12-02 17:28:20.060 -0200 DEBUG: snmpd: Exited 0 times, waiting 0 seconds before respawning. 2019-12-02 17:28:20.061 -0200 DEBUG: snmpd: adding timer deps, event start_helper in 0 seconds 2019-12-02 17:28:20.064 -0200 DEBUG: snmpd: Got event start_helper_event 2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Got timer event deps, removing from active list 2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Calling start_helper_event 2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: transitioning from restarting -> scheduling 2019-12-02 17:28:20.065 -0200 DEBUG: snmpd: Couldn't find cfg node md.apps.s0.mp.grp.websrvr.info 2019-12-02 17:28:20.067 -0200 DEBUG: snmpd: Got event sysdNotif_event [mp-backtrace-snmpd] less mp-backtrace snmpd_9.0.4_39.info ================================================================================ Process snmpd: Mon, 09 Dec 2019 17:41:16 ================================================================================ ------------------------------------------------------------------------------- Backtrace -------------------------------------------------------------------------------- [New LWP 27851] [New LWP 27853] [New LWP 27852] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/snmpd -I-hr_device hr_proc hr_network ifTable ifXTable ipAddressTable'. Program terminated with signal SIGSEGV, Segmentation fault. #0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279 [Current thread is 1 (Thread 0x7f7eaf421740 (LWP 27851))] Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)): #0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279 #1 0x00007f7eaa9e3f84 in update_counter_data () at pan_globalCountersCache.c:509 #2 0x00007f7eaa9cb289 in handle_panGlobalCounters (handler=<optimized out>, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at panSys.c:2279 #3 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c280) at agent_handler.c:526 #4 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c370, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #5 0x00007f7eaefc00d9 in netsnmp_instance_helper_handler (handler=0x7f7eb0e3c370, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/instance.c:775 #6 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c370) at agent_handler.c:526 #7 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c3e0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #8 0x00007f7eaefc2af2 in netsnmp_scalar_helper_handler (handler=0x7f7eb0e3c3e0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/scalar.c:188 #9 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c3e0) at agent_handler.c:526 #10 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c450, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #11 0x00007f7eaefc32ed in netsnmp_serialize_helper_handler (handler=0x7f7eb0e3c450, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=<optimized out>) at helpers/serialize.c:66 #12 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c450) at agent_handler.c:526 #13 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c4c0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #14 0x00007f7eaefbd52a in netsnmp_bulk_to_next_helper (handler=0x7f7eb0e3c4c0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/bulk_to_next.c:110 #15 0x00007f7eaefd03ff in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c4c0) at agent_handler.c:526 #16 netsnmp_call_handlers (reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at agent_handler.c:611 #17 0x00007f7eaefddd79 in handle_var_requests (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:2689 #18 0x00007f7eaefde1c0 in handle_getnext_loop (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3148 #19 0x00007f7eaefdec30 in handle_pdu (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3477 #20 0x00007f7eaefded78 in netsnmp_handle_request (asp=asp@entry=0x7f7eb0ef0800, status=status@entry=0) at snmp_agent.c:3307 #21 0x00007f7eaefdf232 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x7f7eb0eb18b0, magic=<optimized out>) at snmp_agent.c:2000 #22 0x00007f7eae887ae1 in _sess_process_packet (sessp=sessp@entry=0x7f7eb0f0ccb0, sp=sp@entry=0x7f7eb0f0cce0, isp=isp@entry=0x7f7eb0f0c9b0, transport=transport@entry=0x7f7eb0f0ca20, opaque=<optimized out>, olength=<optimized out>, packetptr=packetptr@entry=0x7f7eb0f10220 "01\002\001\001\004\XXXXXXXXXXXXXX\245\037\002\004o\021t\324\002\001", length=51) at snmp_api.c:5437 #23 0x00007f7eae88866e in _sess_read (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5866 #24 0x00007f7eae8894b9 in snmp_sess_read2 (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5898 #25 0x00007f7eae88950b in snmp_read2 (fdset=0x7fffed6f5880) at snmp_api.c:5500 #26 0x00007f7eaf43c7b7 in receive () at snmpd.c:1361 #27 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1140 Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)): #0 0x00007f7ead73b4e9 in __pselect (nfds=11, readfds=readfds@entry=0x7f7eb0d57b80, writefds=writefds@entry=0x7f7eb0d57c80, exceptfds=exceptfds@entry=0x7f7eb0d57d80, timeout=<optimized out>, timeout@entry=0x7f7ea73d2cf0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/pselect.c:77 #1 0x00007f7ea98b9c1e in __evGetNext (opaqueCtx=opaqueCtx@entry=..., opaqueEv=opaqueEv@entry=0x7f7ea73d2ee0, options=options@entry=2) at eventlib.c:328 #2 0x00007f7ea98ba820 in __evMainLoop (opaqueCtx=...) at eventlib.c:654 #3 0x00007f7ea9a0f3fc in sysd_pthread_evloop (arg=0x7f7eb0d54460) at sysd_pthread.c:506 #4 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea73d3700) at pthread_create.c:308 #5 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f7ea9a1d80a in sysd_sched_get_work (sch=0x7f7eb0d63c00, tidx=tidx@entry=0) at sysd_sched.c:172 #2 0x00007f7ea9a0ee5f in sysd_pthread_worker (warg=0x7f7eb0d54460) at sysd_pthread.c:258 #3 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea77d4700) at pthread_create.c:308 #4 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---SYSTEM ARCHITECTURE--- The target architecture is set automatically (currently i386:x86-64) ---SHARED MEMORY GDB--- From To Syms Read Shared Object Library 0x00007f7eaefbc740 0x00007f7eaeffc060 Yes /lib64/libnetsnmpagent.so.31 0x00007f7eaeb69e40 0x00007f7eaec235f4 Yes /lib64/libnetsnmpmibs.so.31 0x00007f7eae85fb80 0x00007f7eae8c972c Yes /lib64/libnetsnmp.so.31 0x00007f7eae642ed0 0x00007f7eae6439d0 Yes /lib64/libdl.so.2 0x00007f7eae3ee5f0 0x00007f7eae427438 Yes /lib64/librpm.so.3 0x00007f7eae1b9fa0 0x00007f7eae1cefe0 Yes /lib64/librpmio.so.3 0x00007f7eadd85900 0x00007f7eadf03af0 Yes /lib64/libcrypto.so.10 0x00007f7eada144b0 0x00007f7eada7e9e8 Yes /lib64/libm.so.6 0x00007f7ead66c3e0 0x00007f7ead7b0670 Yes /lib64/libc.so.6 0x00007f7eaf214ae0 0x00007f7eaf22f2ba Yes /lib64/ld-linux-x86-64.so.2 0x00007f7ead33f550 0x00007f7ead410a84 Yes (*) /lib64/libnss3.so 0x00007f7ead118760 0x00007f7ead1245f0 Yes (*) /lib64/libbz2.so.1 0x00007f7eacf03170 0x00007f7eacf0f6f0 Yes (*) /lib64/libz.so.1 0x00007f7eaccedb70 0x00007f7eaccfb9c8 Yes (*) /lib64/libelf.so.1 0x00007f7eacac8f30 0x00007f7eacadeea0 Yes (*) /lib64/liblzma.so.5 0x00007f7eac8bdcf0 0x00007f7eac8c32a8 Yes (*) /lib64/libpopt.so.0 0x00007f7eac677ca0 0x00007f7eac6a73f4 Yes /lib64/liblua.so 0x00007f7eac2e0cb0 0x00007f7eac420f5c Yes (*) /lib64/libdb-5.3.so 0x00007f7eac09b8a0 0x00007f7eac0a6514 Yes /lib64/libpthread.so.0 0x00007f7eabe75e30 0x00007f7eabe84318 Yes (*) /lib64/libnssutil3.so 0x00007f7eabc66510 0x00007f7eabc67b38 Yes (*) /lib64/libplc4.so 0x00007f7eaba62090 0x00007f7eaba63028 Yes (*) /lib64/libplds4.so 0x00007f7eab82fcd0 0x00007f7eab84fac0 Yes (*) /lib64/libnspr4.so 0x00007f7eab61d2c0 0x00007f7eab6200bc Yes /lib64/librt.so.1 0x00007f7eab40a240 0x00007f7eab411454 Yes /lib64/libnss_files.so.2 0x00007f7eab1ce2d0 0x00007f7eab1f79ac Yes (*) /lib64/libsoftokn3.so 0x00007f7eaaf1fa00 0x00007f7eaafa7860 Yes (*) /lib64/libsqlite3.so.0 0x00007f7eaac98900 0x00007f7eaaced9ad Yes (*) /lib64/libfreeblpriv3.so 0x00007f7eaa9c04d0 0x00007f7eaaa15080 Yes /usr/local/lib64/shobjs/libpanmib.so 0x00007f7eaa79f580 0x00007f7eaa79f668 Yes /lib64/libnetsnmphelpers.so.31 0x00007f7ea97aeb20 0x00007f7ea9a61d10 Yes /usr/local/lib64/libpancommon_mp.so.1 0x00007f7ea95148e0 0x00007f7ea95200fc Yes (*) /lib64/libexslt.so.0 0x00007f7ea92dcc60 0x00007f7ea93010d8 Yes (*) /lib64/libxslt.so.1 0x00007f7ea8f984f0 0x00007f7ea9080514 Yes (*) /lib64/libxml2.so.2 0x00007f7ea8d19870 0x00007f7ea8d520d4 Yes /lib64/libssl.so.10 0x00007f7ea8981120 0x00007f7ea8a60e20 Yes /lib64/libpython2.7.so.1.0 0x00007f7ea86cf9f0 0x00007f7ea8726a4c Yes (*) /lib64/libcurl.so.4 0x00007f7ea84655f0 0x00007f7ea84aa8b0 Yes (*) /lib64/libpcre.so.1 0x00007f7ea8260210 0x00007f7ea8261fe0 Yes /usr/local/lib64/3p/libcjson.so.1 0x00007f7ea805cf10 0x00007f7ea805d804 Yes /lib64/libutil.so.1 0x00007f7ea7e211c0 0x00007f7ea7e4b810 Yes /lib64/libldap-2.4.so.2 0x00007f7ea7c08610 0x00007f7ea7c0f444 Yes /lib64/liblber-2.4.so.2 0x00007f7ea79eea40 0x00007f7ea79fd854 Yes /lib64/libresolv.so.2 0x00007f7ea77d7af0 0x00007f7ea77e7298 Yes (*) /lib64/libgcc_s.so.1 (*): Shared library is missing debugging information. ---THREADS PC--- Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)): => 0x7f7ea9a22e2d <sysd_obj_release+29>: lock subl $0x1,0xc(%rdi) 0x7f7ea9a22e32 <sysd_obj_release+34>: je 0x7f7ea9a22e50 <sysd_obj_release+64> 0x7f7ea9a22e34 <sysd_obj_release+36>: mov 0x28(%rsp),%rax 0x7f7ea9a22e39 <sysd_obj_release+41>: xor %fs:0x28,%rax 0x7f7ea9a22e42 <sysd_obj_release+50>: jne 0x7f7ea9a2303c <sysd_obj_release+556> 0x7f7ea9a22e48 <sysd_obj_release+56>: add $0x30,%rsp 0x7f7ea9a22e4c <sysd_obj_release+60>: pop %rbx 0x7f7ea9a22e4d <sysd_obj_release+61>: retq Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)): => 0x7f7ead73b4e9 <__pselect+169>: cmp $0xfffffffffffff000,%rax 0x7f7ead73b4ef <__pselect+175>: ja 0x7f7ead73b517 <__pselect+215> 0x7f7ead73b4f1 <__pselect+177>: mov %ebp,%edi 0x7f7ead73b4f3 <__pselect+179>: mov %eax,0x8(%rsp) 0x7f7ead73b4f7 <__pselect+183>: callq 0x7f7ead751740 <__libc_disable_asynccancel> 0x7f7ead73b4fc <__pselect+188>: mov 0x8(%rsp),%eax 0x7f7ead73b500 <__pselect+192>: jmp 0x7f7ead73b493 <__pselect+83> 0x7f7ead73b502 <__pselect+194>: mov 0x2cc95f(%rip),%rdx # 0x7f7eada07e68 Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)): => 0x7f7eac0a16d5 <pthread_cond_wait@@GLIBC_2.3.2+197>: mov (%rsp),%edi 0x7f7eac0a16d8 <pthread_cond_wait@@GLIBC_2.3.2+200>: callq 0x7f7eac0a3ea0 <__pthread_disable_asynccancel> 0x7f7eac0a16dd <pthread_cond_wait@@GLIBC_2.3.2+205>: mov 0x8(%rsp),%rdi 0x7f7eac0a16e2 <pthread_cond_wait@@GLIBC_2.3.2+210>: mov $0x1,%esi 0x7f7eac0a16e7 <pthread_cond_wait@@GLIBC_2.3.2+215>: xor %eax,%eax 0x7f7eac0a16e9 <pthread_cond_wait@@GLIBC_2.3.2+217>: lock cmpxchg %esi,(%rdi) 0x7f7eac0a16ed <pthread_cond_wait@@GLIBC_2.3.2+221>: jne 0x7f7eac0a17d3 <pthread_cond_wait@@GLIBC_2.3.2+451> 0x7f7eac0a16f3 <pthread_cond_wait@@GLIBC_2.3.2+227>: mov 0x2c(%rdi),%edx ---THREADS SP--- cfg.platform.portcount: 20 cfg.platform.serial: cfg.platform.total-slots: 1 cfg.platform.variant: base cfg.platform.version: 1.1 cfg.platform.vpn-disable: False cfg.net.s0.eth0.cfg: 4, }, 'up': True, 'v6routes': { 'default': , }, } ha.app.local.state: passive ha.app.peer.state: active -------------------------------------------------------------------------------- Backtrace -------------------------------------------------------------------------------- [New LWP 27851] [New LWP 27853] [New LWP 27852] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/snmpd -I-hr_device hr_proc hr_network ifTable ifXTable ipAddressTable'. Program terminated with signal SIGSEGV, Segmentation fault. #0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279 [Current thread is 1 (Thread 0x7f7eaf421740 (LWP 27851))] Thread 1 (Thread 0x7f7eaf421740 (LWP 27851)): #0 sysd_obj_release (o=0x7f7e00000000) at sysd_obj_api.c:279 #1 0x00007f7eaa9e3f84 in update_counter_data () at pan_globalCountersCache.c:509 #2 0x00007f7eaa9cb289 in handle_panGlobalCounters (handler=<optimized out>, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at panSys.c:2279 #3 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c280) at agent_handler.c:526 #4 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c370, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #5 0x00007f7eaefc00d9 in netsnmp_instance_helper_handler (handler=0x7f7eb0e3c370, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/instance.c:775 #6 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c370) at agent_handler.c:526 #7 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c3e0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #8 0x00007f7eaefc2af2 in netsnmp_scalar_helper_handler (handler=0x7f7eb0e3c3e0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/scalar.c:188 #9 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c3e0) at agent_handler.c:526 #10 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c450, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #11 0x00007f7eaefc32ed in netsnmp_serialize_helper_handler (handler=0x7f7eb0e3c450, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=<optimized out>) at helpers/serialize.c:66 #12 0x00007f7eaefd09fb in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c450) at agent_handler.c:526 #13 netsnmp_call_next_handler (current=current@entry=0x7f7eb0e3c4c0, reginfo=reginfo@entry=0x7f7eb0e3c2f0, reqinfo=reqinfo@entry=0x7f7eb0ea6320, requests=requests@entry=0x7f7eb0ebee50) at agent_handler.c:640 #14 0x00007f7eaefbd52a in netsnmp_bulk_to_next_helper (handler=0x7f7eb0e3c4c0, reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at helpers/bulk_to_next.c:110 #15 0x00007f7eaefd03ff in netsnmp_call_handler (requests=0x7f7eb0ebee50, reqinfo=0x7f7eb0ea6320, reginfo=0x7f7eb0e3c2f0, next_handler=0x7f7eb0e3c4c0) at agent_handler.c:526 #16 netsnmp_call_handlers (reginfo=0x7f7eb0e3c2f0, reqinfo=0x7f7eb0ea6320, requests=0x7f7eb0ebee50) at agent_handler.c:611 #17 0x00007f7eaefddd79 in handle_var_requests (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:2689 #18 0x00007f7eaefde1c0 in handle_getnext_loop (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3148 #19 0x00007f7eaefdec30 in handle_pdu (asp=asp@entry=0x7f7eb0ef0800) at snmp_agent.c:3477 #20 0x00007f7eaefded78 in netsnmp_handle_request (asp=asp@entry=0x7f7eb0ef0800, status=status@entry=0) at snmp_agent.c:3307 #21 0x00007f7eaefdf232 in handle_snmp_packet (op=<optimized out>, session=<optimized out>, reqid=<optimized out>, pdu=0x7f7eb0eb18b0, magic=<optimized out>) at snmp_agent.c:2000 #22 0x00007f7eae887ae1 in _sess_process_packet (sessp=sessp@entry=0x7f7eb0f0ccb0, sp=sp@entry=0x7f7eb0f0cce0, isp=isp@entry=0x7f7eb0f0c9b0, transport=transport@entry=0x7f7eb0f0ca20, opaque=<optimized out>, oleng th=<optimized out>, packetptr=packetptr@entry=0x7f7eb0f10220 "01\002\001\001\004\XXXXXXXXXXXXXXXXXXX\245\037\002\004o\021t\324\002\001", length=51) at snmp_api.c:5437 #23 0x00007f7eae88866e in _sess_read (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5866 #24 0x00007f7eae8894b9 in snmp_sess_read2 (sessp=sessp@entry=0x7f7eb0f0ccb0, fdset=fdset@entry=0x7fffed6f5880) at snmp_api.c:5898 #25 0x00007f7eae88950b in snmp_read2 (fdset=0x7fffed6f5880) at snmp_api.c:5500 #26 0x00007f7eaf43c7b7 in receive () at snmpd.c:1361 #27 main (argc=<optimized out>, argv=<optimized out>) at snmpd.c:1140 Thread 2 (Thread 0x7f7ea73d3700 (LWP 27853)): #0 0x00007f7ead73b4e9 in __pselect (nfds=11, readfds=readfds@entry=0x7f7eb0d57b80, writefds=writefds@entry=0x7f7eb0d57c80, exceptfds=exceptfds@entry=0x7f7eb0d57d80, timeout=<optimized out>, timeout@entry=0x7f7ea73d2cf0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/pselect.c:77 #1 0x00007f7ea98b9c1e in __evGetNext (opaqueCtx=opaqueCtx@entry=..., opaqueEv=opaqueEv@entry=0x7f7ea73d2ee0, options=options@entry=2) at eventlib.c:328 #2 0x00007f7ea98ba820 in __evMainLoop (opaqueCtx=...) at eventlib.c:654 #3 0x00007f7ea9a0f3fc in sysd_pthread_evloop (arg=0x7f7eb0d54460) at sysd_pthread.c:506 #4 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea73d3700) at pthread_create.c:308 #5 0x00007f7ead743ced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f7ea77d4700 (LWP 27852)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f7ea9a1d80a in sysd_sched_get_work (sch=0x7f7eb0d63c00, tidx=tidx@entry=0) at sysd_sched.c:172 #2 0x00007f7ea9a0ee5f in sysd_pthread_worker (warg=0x7f7eb0d54460) at sysd_pthread.c:258 #3 0x00007f7eac09ddc5 in start_thread (arg=0x7f7ea77d4700) at pthread_create.c:308 XXXXXXXXXXXX(passive)> tail follow yes mp-log masterd_detail.log 2019-12-09 18:10:02.639 -0200 DEBUG: stats_gen: exiting from state running from user: False 2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: transitioning from running -> exiting 2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: exit time 1575922202.54, previous start time 1575922201.35 restart_reset 300 2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: restarts 0 max 3 reboot-rule False, restart-en True 2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: not restarting 2019-12-09 18:10:02.640 -0200 DEBUG: stats_gen: transitioning from exiting -> notRunning 2019-12-09 18:10:02.641 -0200 DEBUG: stats_gen: sending childExited event to parent 2019-12-09 18:10:02.644 -0200 DEBUG: batch: Got event childExited_event 2019-12-09 18:10:02.644 -0200 DEBUG: batch: Calling childExited_event 2019-12-09 18:10:02.645 -0200 DEBUG: batch: child stats_gen exited pid 18082   Thanks in advance ... View more