LACP interface ethernet1/24 moved out of AE-group ae1

[Pras]

Hi Guys,

We are getting "LACP interface ethernet1/24 moved out of AE-group ae1" through syslog (emailed) multiple times in a day on PA 3410 running on PAN OS 10.2.3 in HA active/passive. The switch in use is Aruba 8320

Interesting the same msg is received from the passive device too (whereas its interface is in shutdown  mode)

l2ctrld.log has no error message and there is no other error msgs on the system logs. The ports seem to be working fine too.

Below is the last msg from the  l2ctrld.log

The ehmon brdagent logs have no errors related to this port 1/24.

Only thing is that this FW was replaced recently. (although the error started to come only after a month or so from migration). The Switch seems  to  be populating no errors too.

The only other error msgs i see is of "Hardware session Offloading disabled" (Although, I believe this has nothing to do with this LACP port moving out of the group)

Interesting the alerts also do not get disabled: (configuring below has no change in the alert-  I am wondering if this msg is from FW although the alert msg says it's coming from the FW)

Many Thanks,

@S.Cantwell 

@OtakarKlier 

@BPry 

[kiwi]

Hi @Pras ,

Can you check the LACP logs ? Please check if the links are leaving (changing states in) the LAG because of missed LACPDUs from the peer.

This could be a result of the transmission rate mismatch between the peers. Fast mode is very sensitive to the network churn and if no LACPDUs are received for 3 seconds, LACP will go down.

You can set the LACP transmission rate to Slow as to not miss PDUs or LACP updates which is set to a fairly sensitive setting in Fast Mode.

Hope this helps.

-Kiwi.

[Pras]

Hi @kiwi 

Thanks for the reply.

There are no missed LACPDU's and the LACP Tx rate is slow.  Peer is an  Aruba switch.

We receive "

LACP interface ethernet1/24 moved out of AE-group ae1. Selection state Unselected(Link down)

constantly" and strangely there are no logs generated in the Firewall (System/monitor logs)

l2ctrld log has below(the whole pattern in curly bracket is repeats with no other errors in-between), and I wonder if this this is the syncing msg to the passive device.

Interestingly, the passive FW is also throwing the same error where as to the ports (ethernet ports) are not even up. I do not see any bugs of such kind reported either.

Any help will be much appreciated.

Many Thanks,

[BPry]

@Pras,

The alert filter that you have setup under your 'system-critical' entry should definitely be preventing the firewall from sending you alerts about these lacp-up/link-down events. 

Was their a period that the 3410 was functioning without these messages properly, or did they show up as soon as you installed the 3410? 

[Pras]

Hi @BPry 

Thanks for the reply. These were installed on early Dec and were working fine until the start of Jan when the msgs started popping out. I m very surprised that the filters are not blocking the msgs and the errors are coming from the passive FW too (eth ports are not even up).

Many Thanks,

[Pras]

Hi @BPry ,

Even with the Port on the FW shutdown, we are still getting the "ethernet1/24 moved out of AE-group ae1" error. 😞

[BPry]

@Pras,

Just to verify; when you say that you're getting alerts through syslog emailed to you, you simply mean through your log- settings you have it set to email you correct? Do you have the system emailing you directly, or do you pass this to a SIEM and have that emailing you alerts?

I've answered this assuming that it's the first and that you have the firewall emailing you system-critical alerts and not passing those alerts through a SIEM. As long as that's the case, I think you'll have to open a TAC case and see if you aren't running into some sort of weird bug. The fact that you're still getting alerts when you've negated the subtype is just weird, and it shouldn't be happening.

Lastly have you verified that the alerts you're getting are actually present in the system logs? Just verifying that something hasn't gotten "stuck" and keeps resending alerts that the firewall itself isn't actually identifying. 

[Pras]

@BPry 

Yes that is correct, there is no SIEM just the alerts sent directly through email.

There are no alerts in the logs surprisingly. 

[BPry · Accepted solution]

@Pras,

If you aren't seeing the associated log on the device itself, it sounds like something with the log-receiver process is just continually stuck processing. You can try restarting that process itself via the 'debug software restart process log-receiver' and seeing if that clears things up if you haven't tried that already.

I've assumed that you've tried restarting both units to see if it clears things, but if not that would be my next step. If neither of those things work I'd definitely pass this to TAC to help troubleshoot. You shouldn't be getting email notices if you don't have an associated system log, so something is definitely not being processed properly. 

[Pras]

@BPry 

Reboot resolved the issue 😄

Thanks a lot for your help and suggestions