BGP neighbor drop

[Alpalo]

Hello,

Model: PA-5260
Version: 9.1.7
The device has lost the connection against all the BGP neighbors that are connected through interface ae3.

The swtich where the interfaces connect has also lost the connection against the BGP neighbors and also does not show in the logs any failure of the interfaces.

Checking the qtrace_routed.log file I have found the following logs.... Any idea what could have caused this crash?

30766 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1168] locla ip X.X.X.153 remote ip X.X.X.154 state change 3 => 1, prev state 3 if 166
21530767 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1173] locla ip X.X.X.153 remote ip X.X.X.154 local diag 3 remote diag 1, agent dp 0 correlator 171
21530768 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1168] locla ip X.X.X.137 remote ip X.X.X.139 state change 3 => 1, prev state 3 if 163
21530769 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1173] locla ip X.X.X.137 remote ip X.X.X.139 local diag 3 remote diag 1, agent dp 0 correlator 167
21530770 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1168] locla ip X.X.X.121 remote ip X.X.X.122 state change 3 => 1, prev state 3 if 157
21530771 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1173] locla ip X.X.X.121 remote ip X.X.X.122 local diag 3 remote diag 1, agent dp 0 correlator 140
21530772 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1168] locla ip X.X.X.161 remote ip X.X.X.163 state change 3 => 1, prev state 3 if 171
21530773 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1173] locla ip X.X.X.161 remote ip X.X.X.163 local diag 3 remote diag 1, agent dp 0 correlator 172
21530774 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1168] locla ip X.X.X.25 remote ip X.X.X.27 state change 3 => 1, prev state 3 if 135
21530775 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1173] locla ip X.X.X.25 remote ip X.X.X.27 local diag 3 remote diag 1, agent dp 0 correlator 153
21530776 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1168] locla ip X.X.X.137 remote ip X.X.X.138 state change 3 => 1, prev state 3 if 163
21530777 [2022-09-08 11:03:48.0915] [pan_dc_process_bfd_event:1173] locla ip X.X.X.137 remote ip X.X.X.138 local diag 3 remote diag 1, agent dp 0 correlator 166
21530778 [2022-09-08 11:03:48.0916] [pan_dc_process_bfd_event:1168] locla ip X.X.X.57 remote ip X.X.X.58 state change 3 => 1, prev state 3 if 151
21530779 [2022-09-08 11:03:48.0916] [pan_dc_process_bfd_event:1173] locla ip X.X.X.57 remote ip X.X.X.58 local diag 3 remote diag 1, agent dp 0 correlator 147
21530780 [2022-09-08 11:03:48.0917] [pan_dc_process_bfd_event:1168] locla ip X.X.X.145 remote ip X.X.X.146 state change 3 => 1, prev state 3 if 164
21530781 [2022-09-08 11:03:48.0917] [pan_dc_process_bfd_event:1173] locla ip X.X.X.145 remote ip X.X.X.146 local diag 3 remote diag 1, agent dp 0 correlator 168
21530782 [2022-09-08 11:03:48.0918] [pan_dc_process_bfd_event:1168] locla ip X.X.X.161 remote ip X.X.X.162 state change 3 => 1, prev state 3 if 171
21530783 [2022-09-08 11:03:48.0918] [pan_dc_process_bfd_event:1173] locla ip X.X.X.161 remote ip X.X.X.162 local diag 3 remote diag 1, agent dp 0 correlator 173
21530784 [2022-09-08 11:03:48.0919] [pan_dc_process_bfd_event:1168] locla ip X.X.X.17 remote ip X.X.X.18 state change 3 => 1, prev state 3 if 134
21530785 [2022-09-08 11:03:48.0919] [pan_dc_process_bfd_event:1173] locla ip X.X.X.17 remote ip X.X.X.18 local diag 3 remote diag 1, agent dp 0 correlator 143
21530786 [2022-09-08 11:03:48.0920] [pan_dc_process_bfd_event:1168] locla ip X.X.X.97 remote ip X.X.X.98 state change 3 => 1, prev state 3 if 147
21530787 [2022-09-08 11:03:48.0920] [pan_dc_process_bfd_event:1173] locla ip X.X.X.97 remote ip X.X.X.98 local diag 3 remote diag 1, agent dp 0 correlator 149
21530788 [2022-09-08 11:03:48.0921] [pan_dc_process_bfd_event:1168] locla ip X.X.X.145 remote ip X.X.X.147 state change 3 => 1, prev state 3 if 164
21530789 [2022-09-08 11:03:48.0921] [pan_dc_process_bfd_event:1173] locla ip X.X.X.145 remote ip X.X.X.147 local diag 3 remote diag 1, agent dp 0 correlator 169
21530790 [2022-09-08 11:03:48.0922] [pan_dc_process_bfd_event:1168] locla ip X.X.X.153 remote ip X.X.X.155 state change 3 => 1, prev state 3 if 166
21530791 [2022-09-08 11:03:48.0922] [pan_dc_process_bfd_event:1173] locla ip X.X.X.153 remote ip X.X.X.155 local diag 3 remote diag 1, agent dp 0 correlator 170
21530792 [2022-09-08 11:03:48.0923] [pan_dc_process_bfd_event:1168] locla ip X.X.X.73 remote ip X.X.X.74 state change 3 => 1, prev state 3 if 155
21530793 [2022-09-08 11:03:48.0923] [pan_dc_process_bfd_event:1173] locla ip X.X.X.73 remote ip X.X.X.74 local diag 3 remote diag 1, agent dp 0 correlator 148
21530794 [2022-09-08 11:03:48.0924] [pan_dc_process_bfd_event:1168] locla ip X.X.X.105 remote ip X.X.X.106 state change 3 => 1, prev state 3 if 149
21530795 [2022-09-08 11:03:48.0924] [pan_dc_process_bfd_event:1173] locla ip X.X.X.105 remote ip X.X.X.106 local diag 3 remote diag 1, agent dp 0 correlator 150
21530796 [2022-09-08 11:03:48.0926] [pan_dc_process_bfd_event:1168] locla ip X.X.X.129 remote ip X.X.X.131 state change 3 => 1, prev state 3 if 159
21530797 [2022-09-08 11:03:48.0926] [pan_dc_process_bfd_event:1173] locla ip X.X.X.129 remote ip X.X.X.131 local diag 3 remote diag 1, agent dp 0 correlator 161
21530798 [2022-09-08 11:03:48.0927] [pan_dc_process_bfd_event:1168] locla ip X.X.X.49 remote ip X.X.X.51 state change 3 => 1, prev state 3 if 143
21530799 [2022-09-08 11:03:48.0927] [pan_dc_process_bfd_event:1173] locla ip X.X.X.49 remote ip X.X.X.51 local diag 3 remote diag 1, agent dp 0 correlator 155
21530800 [2022-09-08 11:03:48.0928] [pan_dc_process_bfd_event:1168] locla ip X.X.X.57 remote ip X.X.X.59 state change 3 => 1, prev state 3 if 151
21530801 [2022-09-08 11:03:48.0928] [pan_dc_process_bfd_event:1173] locla ip X.X.X.57 remote ip X.X.X.59 local diag 3 remote diag 1, agent dp 0 correlator 156
21530802 [2022-09-08 11:03:48.0934] [pan_dc_process_bfd_event:1168] locla ip X.X.X.97 remote ip X.X.X.99 state change 3 => 1, prev state 3 if 147
21530803 [2022-09-08 11:03:48.0934] [pan_dc_process_bfd_event:1173] locla ip X.X.X.97 remote ip X.X.X.99 local diag 3 remote diag 1, agent dp 0 correlator 159
21530804 [2022-09-08 11:03:48.0935] [pan_dc_process_bfd_event:1168] locla ip X.X.X.17 remote ip X.X.X.19 state change 3 => 1, prev state 3 if 134
21530805 [2022-09-08 11:03:48.0935] [pan_dc_process_bfd_event:1173] locla ip X.X.X.17 remote ip X.X.X.19 local diag 3 remote diag 1, agent dp 0 correlator 152
21530806 [2022-09-08 11:03:48.0936] [pan_dc_process_bfd_event:1168] locla ip X.X.X.49 remote ip X.X.X.50 state change 3 => 1, prev state 3 if 143
21530807 [2022-09-08 11:03:48.0936] [pan_dc_process_bfd_event:1173] locla ip X.X.X.49 remote ip X.X.X.50 local diag 3 remote diag 1, agent dp 0 correlator 146
21530808 [2022-09-08 11:03:48.0937] [pan_dc_process_bfd_event:1168] locla ip X.X.X.121 remote ip X.X.X.123 state change 3 => 1, prev state 3 if 157
21530809 [2022-09-08 11:03:48.0937] [pan_dc_process_bfd_event:1173] locla ip X.X.X.121 remote ip X.X.X.123 local diag 3 remote diag 1, agent dp 0 correlator 141
21530810 [2022-09-08 11:03:48.0937] [pan_dc_process_bfd_event:1168] locla ip X.X.X.129 remote ip X.X.X.130 state change 3 => 1, prev state 3 if 159
21530811 [2022-09-08 11:03:48.0937] [pan_dc_process_bfd_event:1173] locla ip X.X.X.129 remote ip X.X.X.130 local diag 3 remote diag 1, agent dp 0 correlator 162
21530812 [2022-09-08 11:03:48.0939] [pan_dc_process_bfd_event:1168] locla ip X.X.X.105 remote ip X.X.X.107 state change 3 => 1, prev state 3 if 149
21530813 [2022-09-08 11:03:48.0939] [pan_dc_process_bfd_event:1173] locla ip X.X.X.105 remote ip X.X.X.107 local diag 3 remote diag 1, agent dp 0 correlator 160
21530814 [2022-09-08 11:03:49.0161] [pan_dc_process_bfd_event:1168] locla ip X.X.X.25 remote ip X.X.X.26 state change 3 => 1, prev state 3 if 135
21530815 [2022-09-08 11:03:49.0161] [pan_dc_process_bfd_event:1173] locla ip X.X.X.25 remote ip X.X.X.26 local diag 1 remote diag 0, agent dp 0 correlator 144
21530816 [2022-09-08 11:03:49.0161] [pan_dc_process_bfd_event:1168] locla ip X.X.X.41 remote ip X.X.X.43 state change 3 => 1, prev state 3 if 142
21530817 [2022-09-08 11:03:49.0161] [pan_dc_process_bfd_event:1173] locla ip X.X.X.41 remote ip X.X.X.43 local diag 1 remote diag 0, agent dp 0 correlator 154
21530818 [2022-09-08 11:03:49.0162] [pan_dc_process_bfd_event:1168] locla ip X.X.X.73 remote ip X.X.X.75 state change 3 => 1, prev state 3 if 155
21530819 [2022-09-08 11:03:49.0162] [pan_dc_process_bfd_event:1173] locla ip X.X.X.73 remote ip X.X.X.75 local diag 1 remote diag 0, agent dp 0 correlator 157
21530820 [2022-09-08 11:03:49.0162] [pan_dc_process_bfd_event:1168] locla ip X.X.X.41 remote ip X.X.X.42 state change 3 => 1, prev state 3 if 142
21530821 [2022-09-08 11:03:49.0162] [pan_dc_process_bfd_event:1173] locla ip X.X.X.41 remote ip X.X.X.42 local diag 1 remote diag 0, agent dp 0 correlator 145
21530822 [2022-09-08 11:03:49.0163] [pan_dc_process_bfd_event:1168] locla ip X.X.X.65 remote ip X.X.X.66 state change 3 => 1, prev state 3 if 153
21530823 [2022-09-08 11:03:49.0163] [pan_dc_process_bfd_event:1173] locla ip X.X.X.65 remote ip X.X.X.66 local diag 1 remote diag 0, agent dp 0 correlator 163
21530824 [2022-09-08 11:03:49.0164] [pan_dc_process_bfd_event:1168] locla ip X.X.X.65 remote ip X.X.X.67 state change 3 => 1, prev state 3 if 153
21530825 [2022-09-08 11:03:49.0164] [pan_dc_process_bfd_event:1173] locla ip X.X.X.65 remote ip X.X.X.67 local diag 1 remote diag 0, agent dp 0 correlator 158

Regards

[PavelK]

Hello @Alpalo

from this log, it looks like you are running BFD (Bidirectional Forwarding Detection) and this has failed and signaled to take BGP session down. From the "local diag 3" I assume it is a diagnostic code: "3—Neighbor Signaled Session Down". Below is a link with all the codes:

https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-networking-admin/bfd/reference-bfd-details

Kind Regards

Pavel

[Alpalo]

@PavelK 

Thanks so much for your answer, then you understand that the problem is in the neighbor, right?

Regards

[PavelK]

Thank you for reply @Alpalo

based on this log, I would say yes, it was signaled from the neighbor.

I would like to add that the last a few entries in the log look slightly different. These neighbor IP addresses: X.X.X.26, X.X.X.43, X.X.X.75, X.X.X.42, X.X.X.66, X.X.X.67 have log: "local diag 1" which is: "1—Control Detection Time Expired". These BGP sessions were taken down after BFD timer expired. This however does not change fact that this is most likely other side issue.

I looks like you do not have access to other device logs. If you have a chance to check logs, it would be helpful to understand what actually happened. 

Kind Regards

Pavel