Disconnected from Log collector Server

[MP18]

Tonight we got email alerts that our firewalls are disonncted from the log collecors-M500

Below is ms log from the PA

2019-04-05 01:38:55.024 -0600 MS: disconnected from log-collector. waitcount=1
2019-04-05 01:38:55.024 -0600 lcs agent: channel teardown (to 10.7.1.139) complete.
2019-04-05 01:38:55.035 -0600 Error: pan_conn_ext_send_base(cs_conn.c:2601): connmgr: send failure. no conn entry: devid=log-collector
2019-04-05 01:38:55.035 -0600 Error: pan_cfg_log_buffer_nsend(pan_cfg_log_buffer.c:172): Failed to send the logrec to log collector
2019-04-05 01:38:55.035 -0600 Error: pan_log_buffer_cursor_next(pan_cfg_log_buffer.c:1538): logbuffer: failed to send 10 'system' logs to cms
2019-04-05 01:39:00.025 -0600 COMM: connection established. sock=50 remote ip=10.7.1.139 port=3978 local port=47854
2019-04-05 01:39:00.025 -0600 lcs agent: Pre. send buffer limit=22600. s=50
2019-04-05 01:39:00.025 -0600 lcs agent: Post. send buffer limit=1048576. s=50
2019-04-05 01:39:02.057 -0600 lcs agent: ssl channel established. sock=50 ssl=0x121a3400
2019-04-05 01:39:02.058 -0600 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:372): failed to fetch cfg.saas.custid
2019-04-05 01:39:02.059 -0600 Error: pan_get_current_gp_datafile_release_date(pan_cfg_utils.c:5526): Failed to parse file /opt/pancfg/mgmt/global-protect/av-data/av_data_file.dat
2019-04-05 01:39:02.179 -0600 lcs agent: registration request sent. len=29215 sock=50
2019-04-05 01:39:02.260 -0600 connmgr: connection entry added: devid=log-collector sock=50, clientid=0
2019-04-05 01:39:02.261 -0600 connected to Log Collector 007307001117 (key log-collector)
2019-04-05 01:39:02.278 -0600 received a log-fwd-ctrl(start-from-lastack) message from panorama
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[0] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[1] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[2] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[3] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[4] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[5] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[6] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[7] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[8] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[9] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[10] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[11] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[12] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[13] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[14] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[15] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[16] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[17] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[18] LastAck
2019-04-05 01:39:02.278 -0600 Logbuffer: start_from[19] LastAck
2019-04-05 01:39:02.484 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:02.484 -0600 latest logid for hipmatch: 0, recvtime: 0
2019-04-05 01:39:02.484 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:21.651 -0600 latest logid for userid: 0, recvtime: 0
2019-04-05 01:39:21.652 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:21.652 -0600 latest logid for gtp: 0, recvtime: 0
2019-04-05 01:39:21.653 -0600 Error: pan_log_query_expr_get_time_envelope(pan_logdb.c:2598): failed to determine time envelope
2019-04-05 01:39:21.653 -0600 latest logid for auth: 0, recvtime: 0
2019-04-05 01:40:07.274 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:40:42.276 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:41:17.282 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:41:52.285 -0600 Error: pan_conn_entry_write_lock_timeout(cs_conn.c:4227): connmgr: get wr lock timeout. result=110
2019-04-05 01:42:22.024 -0600 Error: csSendWithTimeoutChunk(cs_comm_utils.c:443): COMMS: sock=50. SSL write error fatal. code=5 error=Broken pipe(32) retrycount=0 len=8052 remain=8052 sent=-1
2019-04-05 01:42:22.024 -0600 Error: cs_msg_tcp_send_ex(cs_transport.c:165): COMM: failed to send payload. result=0 len=8052 ctype=3 dtype=1 mtype=0 sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 Device lcs agent log-collector disconnected
2019-04-05 01:42:22.024 -0600 connmgr: shutdown channel. sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 connmgr: connection entry not removed for pending refcount. devid=log-collector cesock=50 sockfd=4294967295 refcount=1
2019-04-05 01:42:22.024 -0600 Error: pan_cfg_log_buffer_nsend(pan_cfg_log_buffer.c:172): Failed to send the logrec to log collector
2019-04-05 01:42:22.024 -0600 Error: pan_log_buffer_cursor_next(pan_cfg_log_buffer.c:1538): logbuffer: failed to send 10 'traffic' logs to cms
2019-04-05 01:42:22.024 -0600 COMMS: ssl write - shutdown exit. sock=50 err=0 sslerr=1 errnum=336396495(protocol is shutdown)
2019-04-05 01:42:22.024 -0600 Error: cs_msg_tcp_send_ex(cs_transport.c:154): COMM: failed to send header. result=0 len=12 ctype=3 dtype=7 mtype=0 sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 Error: pan_lcsa_tcp_channel_loop(src_panos/lcs_agent.c:2678): lcs agent: failed send probe. tcp send failure. sock=50 ssl=0x121a3400
2019-04-05 01:42:22.024 -0600 connmgr: connection entry removed. devid=log-collector sock=50 result=0
2019-04-05 01:42:22.024 -0600 connmgr: unlock - remove conn entry. devid=log-collector sock=50 result=0
2019-04-05 01:42:22.024 -0600 lcs agent: peer watch. sock=50 curtime=25052064 recvtime=25051889 proctime=25051889 sendtime=25052064 errcount=1
2019-04-05 01:42:22.024 -0600 COMMS: ssl read - zero byte. sock=50 err=0 sslerr=6 errnum=0((null))
2019-04-05 01:42:22.024 -0600 Error: cs_recv_tcp_data(cs_comm_utils.c:641): COMMS: could not read header. sock=50 ssl=0x121a3400 len=-1 hdrlen=0

[BPry]

@MP18,

Based off the logs it looks like the connection to 10.7.1.139 is spotty and the firewall can't form a good connection to the log collector. Is this working now or has it stayed disconnected? 

[MP18]

now all the firewalls are connected to log collector.

this happended few times tonight.

is this physical connection issue or some bug?

[MP18]

i checked the network each device and interface between PA and log collectores no issues.

[MP18]

Seems we are only using management interface of m500.

How can i use other interfaces of m500 to collect logs from the firewalls?

[BPry · Accepted solution]

@MP18,

Best practice on this would to actually seperate out interfaces for log collection, and a seperate one for collector group comms. Depending on how many devices you have communicating to the M500 you actually might benefit from having multiple interfaces configured from log collection, not just a single interface. 

Documentation on the Log Collector interface settings can be found starting HERE in the documentation. 

[MP18]

Will do this now going forward.

Thanks for confirming that.