Disconnected from Log collector Server

MP18 — Fri, 05 Apr 2019 13:00:57 GMT

Tonight we got email alerts that our firewalls are disonncted from the log collecors-M500

Below is ms log from the PA

2019-04-05 01:38:55.024 -0600 MS: disconnected from log-collector. waitcount=1
2019-04-05 01:38:55.024 -0600 lcs agent: channel teardown (to 10.7.1.139) complete.
2019-04-05 01:38:55.035 -0600 Error: pan_conn_ext_send_base(cs_conn.c:2601): connmgr: send failure. no conn entry: devid=log-collector
2019-04-05 01:38:55.035 -0600 Error: pan_cfg_log_buffer_nsend(pan_cfg_log_buffer.c:172): Failed to send the logrec to log collector
2019-04-05 01:38:55.035 -0600 Error: pan_log_buffer_cursor_next(pan_cfg_log_buffer.c:1538): logbuffer: failed to send 10 'system' logs to cms
2019-04-05 01:39:00.025 -0600 COMM: connection established. sock=50 remote ip=10.7.1.139 port=3978 local port=47854
2019-04-05 01:39:00.025 -0600 lcs agent: Pre. send buffer limit=22600. s=50
2019-04-05 01:39:00.025 -0600 lcs agent: Post. send buffer limit=1048576. s=50
2019-04-05 01:39:02.057 -0600 lcs agent: ssl channel established. sock=50 ssl=0x121a3400
2019-04-05 01:39:02.058 -0600 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:372): failed to fetch cfg.saas.custid
2019-04-05 01:39:02.059 -0600 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5526): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2019-04-05 01:39:02.179 -0600 lcs agent: registration request sent. len=29215 sock=50
2019-04-05 01:39:02.260 -0600 connmgr: connection entry added: devid=log-collector sock=50, clientid=0
2019-04-05 01:39:02.261 -0600 connected to Log Collector 007307001117 (key log-collector)
2019-04-05 01:39:02.278 -0600 received a log-fwd-ctrl(start-from-lastack) message from panorama
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[0] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[1] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[2] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[3] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[4] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[5] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[6] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[7] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[8] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[9] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[10] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[11] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[12] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[13] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[14] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[15] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[16] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[17] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[18] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[19] LastAck
2019-04-05 01:39:02.484 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:02.484 -0600 latest logid for hipmatch: 0, recvtime: 0
2019-04-05 01:39:02.484 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:21.651 -0600 latest logid for userid: 0, recvtime: 0
2019-04-05 01:39:21.652 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:21.652 -0600 latest logid for gtp: 0, recvtime: 0
2019-04-05 01:39:21.653 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:21.653 -0600 latest logid for auth: 0, recvtime: 0
2019-04-05 01:40:07.274 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:40:42.276 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:41:17.282 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:41:52.285 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:42:22.024 -0600 Error: csSendWithTimeoutChunk(cs_comm_utils.c:443): COMMS: sock=50. SSL write error fatal. code=5 error=Broken pipe(32) retrycount=0 len=8052 remain=8052 sent=-1
2019-04-05 01:42:22.024 -0600 Error: cs_msg_tcp_send_ex(cs_transport.c:165): COMM: failed to send payload. result=0 len=8052 ctype=3 dtype=1 mtype=0 sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 Device lcs agent log-collector disconnected
2019-04-05 01:42:22.024 -0600 connmgr: shutdown channel. sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 connmgr: connection entry not removed for pending refcount. devid=log-collector cesock=50 sockfd=4294967295 refcount=1
2019-04-05 01:42:22.024 -0600 Error: pan_cfg_log_buffer_nsend(pan_cfg_log_buffer.c:172): Failed to send the logrec to log collector
2019-04-05 01:42:22.024 -0600 Error: pan_log_buffer_cursor_next(pan_cfg_log_buffer.c:1538): logbuffer: failed to send 10 'traffic' logs to cms
2019-04-05 01:42:22.024 -0600 COMMS: ssl write - shutdown exit. sock=50 err=0 sslerr=1 errnum=336396495(protocol is shutdown)
2019-04-05 01:42:22.024 -0600 Error: cs_msg_tcp_send_ex(cs_transport.c:154): COMM: failed to send header. result=0 len=12 ctype=3 dtype=7 mtype=0 sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 Error: pan_lcsa_tcp_channel_loop(src_panos/lcs_agent.c:2678): lcs agent: failed send probe. tcp send failure. sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 connmgr: connection entry removed. devid=log-collector sock=50 result=0
2019-04-05 01:42:22.024 -0600 connmgr: unlock - remove conn entry. devid=log-collector sock=50 result=0
2019-04-05 01:42:22.024 -0600 lcs agent: peer watch. sock=50 curtime=25052064 recvtime=25051889 proctime=25051889 sendtime=25052064 errcount=1
2019-04-05 01:42:22.024 -0600 COMMS: ssl read - zero byte. sock=50 err=0 sslerr=6 errnum=0((null))
2019-04-05 01:42:22.024 -0600 Error: cs_recv_tcp_data(cs_comm_utils.c:641): COMMS: could not read header. sock=50 ssl=0x121a3400 len=-1 hdrlen=0

Re: Disconnected from Log collector Server

BPry — Fri, 05 Apr 2019 19:48:30 GMT

@MP18,

Based off the logs it looks like the connection to 10.7.1.139 is spotty and the firewall can't form a good connection to the log collector. Is this working now or has it stayed disconnected?

Re: Disconnected from Log collector Server

MP18 — Fri, 05 Apr 2019 21:50:59 GMT

now all the firewalls are connected to log collector.

this happended few times tonight.

is this physical connection issue or some bug?

Re: Disconnected from Log collector Server

MP18 — Sat, 06 Apr 2019 16:34:07 GMT

i checked the network each device and interface between PA and log collectores no issues.

Re: Disconnected from Log collector Server

MP18 — Sat, 06 Apr 2019 22:21:11 GMT

Seems we are only using management interface of m500.

How can i use other interfaces of m500 to collect logs from the firewalls?

Re: Disconnected from Log collector Server

BPry — Sun, 07 Apr 2019 01:56:27 GMT

@MP18,

Best practice on this would to actually seperate out interfaces for log collection, and a seperate one for collector group comms. Depending on how many devices you have communicating to the M500 you actually might benefit from having multiple interfaces configured from log collection, not just a single interface.

Documentation on the Log Collector interface settings can be found starting HERE in the documentation.

Re: Disconnected from Log collector Server

MP18 — Sun, 07 Apr 2019 02:52:11 GMT

Will do this now going forward.

Thanks for confirming that.

topic Re: Disconnected from Log collector Server in General Topics

Disconnected from Log collector Server

Re: Disconnected from Log collector Server

Re: Disconnected from Log collector Server

Re: Disconnected from Log collector Server

Re: Disconnected from Log collector Server

Re: Disconnected from Log collector Server

Re: Disconnected from Log collector Server