07-30-2024 01:10 AM
Hi Team,
Reaching out for help to identify the main cause of this problem.
I can see that the OSPF adjacency is flapping continuously and I have no clue how to move further and how to identify the cause.
While collecting the routed.log, I can see below:
**** AUDIT 0x0309 - 57 (0001) **** I:0087de60 F:00000002
i3emuif2.c 484 :at 15:13:10, 30 July 2024 (2146345217 ms)
The sub layer oper status of i/f 0X00000114 changed from 0X01 to 0X02.
**** AUDIT 0x0309 - 56 (0002) **** I:0087de60 F:00000002
i3emuif2.c 503 :at 15:13:10, 30 July 2024 (2146345217 ms)
The operational status of interface 0X00000114 has changed from 0X01 to 0X02.
**** AUDIT 0x0309 - 57 (0001) **** I:0087de64 F:00000002
i3emuif2.c 484 :at 15:13:10, 30 July 2024 (2146345217 ms)
The sub layer oper status of i/f 0X00000114 changed from 0X01 to 0X02.
**** AUDIT 0x0309 - 56 (0002) **** I:0087de64 F:00000002
i3emuif2.c 503 :at 15:13:10, 30 July 2024 (2146345217 ms)
The operational status of interface 0X00000114 has changed from 0X01 to 0X02.
**** AUDIT 0x3e01 - 91 (0000) **** I:0087de66 F:00000002
qodmnmi.c 215 :at 15:13:10, 30 July 2024 (2146345217 ms)
OSPF 1 An adjacency with a neighbor has gone down.
Resources associated with database exchange for this neighbor will be
freed.
Neighbor router ID 172.29.254.145
Neighbor IP address 172.29.254.197
Interface category network interface
Interface neighbor IP addr 172.29.254.197 i/f idx 0X00000000
**** AUDIT 0x3e01 - 210 (0000) **** I:0087de66 F:00000002
qoamnfsa.c 768 :at 15:13:10, 30 July 2024 (2146345217 ms)
OSPF 1 i/f idx 0X00000114 rtr ID 172.29.254.145 IP addr 172.29.254.197 neighbor FSM state has deteriorated.
100% Interface address = IP addr 172.29.254.198
OSPF link category = 1
Is neighbor virtual? = 0
FSM input = QOAM_NBR_KILL_NBR (12)
Old FSM state = AMB_OSPF_NBR_FULL (8)
New FSM state = AMB_OSPF_NBR_DOWN (1)
FSM action = H (8)
Neighbor friend status = 1
Number of neighbor events = 6
Number of database exchange timeouts = 0
**** AUDIT 0x0303 - 76 (0000) **** I:0087de6a F:00000002
sckutil.c 1612 :at 15:13:10, 30 July 2024 (2146345217 ms)
Closed a socket
Socket ID = 1026
Socket type = 0X00000003
Socket family = 0X00000002
Socket protocol = 0X000000F8
2024-07-30 15:13:13.496 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:13:15.744 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:13:15.744 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:13:15.746 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:13:27.445 +0800 *PWC* ifmon: pan_routed_ifp_oper_status_tn up(0) for tunnel if 276 vr 1, v6 0, multicast 0
**** AUDIT 0x0309 - 57 (0001) **** I:0087deb2 F:00000002
i3emuif2.c 484 :at 15:13:27, 30 July 2024 (2146361927 ms)
The sub layer oper status of i/f 0X00000114 changed from 0X02 to 0X01.
**** AUDIT 0x0309 - 56 (0002) **** I:0087deb2 F:00000002
100% i3emuif2.c 503 :at 15:13:27, 30 July 2024 (2146361927 ms)
The operational status of interface 0X00000114 has changed from 0X02 to 0X01.
**** AUDIT 0x0309 - 57 (0001) **** I:0087deb6 F:00000002
i3emuif2.c 484 :at 15:13:27, 30 July 2024 (2146361927 ms)
The sub layer oper status of i/f 0X00000114 changed from 0X02 to 0X01.
**** AUDIT 0x0309 - 56 (0002) **** I:0087deb6 F:00000002
i3emuif2.c 503 :at 15:13:27, 30 July 2024 (2146361927 ms)
The operational status of interface 0X00000114 has changed from 0X02 to 0X01.
2024-07-30 15:13:32.454 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:13:32.454 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:13:32.456 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:13:37.454 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:13:37.454 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:13:37.456 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
**** AUDIT 0x3e01 - 200 (0000) **** I:0087dfd0 F:00000002
qoamddsc.c 1053 :at 15:14:03, 30 July 2024 (2146397758 ms)
OSPF 1 Database exchange with an adjacent OSPF neighbor has been completed.
The adjacency with this neighbor is now fully up.
Neighbor router ID 172.29.254.145
Neighbor IP address 172.29.254.197
Interface category network interface
Interface neighbor IP addr 172.29.254.197 i/f idx 0X00000000
2024-07-30 15:14:03.684 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:14:03.684 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:14:03.686 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
100% 2024-07-30 15:14:12.913 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:14:12.914 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:14:12.916 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:14:13.996 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:15:14.008 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
**** AUDIT 0x3e01 - 91 (0000) **** I:0087e170 F:00000002
qodmnmi.c 215 :at 15:15:33, 30 July 2024 (2146488209 ms)
OSPF 1 An adjacency with a neighbor has gone down.
Resources associated with database exchange for this neighbor will be
freed.
Neighbor router ID 172.29.254.145
Neighbor IP address 172.29.254.197
Interface category network interface
Interface neighbor IP addr 172.29.254.197 i/f idx 0X00000000
**** AUDIT 0x3e01 - 210 (0000) **** I:0087e170 F:00000002
qoamnfsa.c 768 :at 15:15:33, 30 July 2024 (2146488209 ms)
OSPF 1 i/f idx 0X00000114 rtr ID 172.29.254.145 IP addr 172.29.254.197 neighbor FSM state has deteriorated.
Interface address = IP addr 172.29.254.198
OSPF link category = 1
Is neighbor virtual? = 0
FSM input = QOAM_NBR_1WAY_RCVD (11)
Old FSM state = AMB_OSPF_NBR_FULL (8)
New FSM state = AMB_OSPF_NBR_INIT (3)
FSM action = K (11)
Neighbor friend status = 1
Number of neighbor events = 6
Number of database exchange timeouts = 0
100% 2024-07-30 15:15:38.754 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:15:38.754 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:15:38.756 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
error: cannot stat /etc/logrotate.d/frr: No such file or directory
2024-07-30 15:16:03.091 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:16:03.091 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:16:03.094 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:16:14.506 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
**** AUDIT 0x3e01 - 200 (0000) **** I:0087e2d4 F:00000002
qoamddsc.c 1053 :at 15:16:33, 30 July 2024 (2146547890 ms)
OSPF 1 Database exchange with an adjacent OSPF neighbor has been completed.
The adjacency with this neighbor is now fully up.
Neighbor router ID 172.29.254.145
Neighbor IP address 172.29.254.197
Interface category network interface
Interface neighbor IP addr 172.29.254.197 i/f idx 0X00000000
2024-07-30 15:16:38.424 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:16:38.424 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:16:38.426 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:16:48.724 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:16:48.724 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:16:48.726 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:17:15.006 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:18:15.495 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:19:15.506 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:20:15.996 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
error: cannot stat /etc/logrotate.d/frr: No such file or directory
2024-07-30 15:21:15.996 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
100% 2024-07-30 15:22:02.814 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:22:02.814 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:22:02.816 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:22:16.495 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:23:16.496 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:24:16.996 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:25:17.495 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
error: cannot stat /etc/logrotate.d/frr: No such file or directory
2024-07-30 15:25:55.874 +0800 TM_SPF: start full SPF calculation rid 172.29.254.114
2024-07-30 15:25:55.874 +0800 TM_SPF: Do the full SPF calculation rid 172.29.254.114
2024-07-30 15:25:55.876 +0800 TM_SPF: full routing calculation finished rid 172.29.254.114
2024-07-30 15:26:17.496 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:27:17.995 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:28:17.996 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
2024-07-30 15:29:18.016 +0800 IFMon: refresh ifstates, server path sw.dev.runtime.
