topic Re: Several sessions outage/interface drops in General Topics

Several sessions outage/interface drops

soporteseguridad — Thu, 20 Jul 2017 14:10:44 GMT

Hi,

We have a PA-5050 with PaNoS 7.0.7, we are expecting that there are moments during the day when the traffic increases that there is a outage for several sessions, but the sessions are still very low for this PA-5050. And we dont know why some sessions are not being established.

We would like to discard if this is caused for PA. Looking the PA interface. We have an ae with 4 ports. We see packet drops in this ae1, the packets dropped are incresing all the time.

On the another hand, if we check all the interfaces (ethernet1/21 ethernet1/22 ethernet1/23 ethernet1/24) bonding in this ae, we dont see any errors.

show interface ae1

--------------------------------------------------------------------------------
Name: ae1, ID: 48
Link status:
Runtime link speed/duplex/state: [n/a]/[n/a]/up
Configured link speed/duplex/state: auto/auto/auto
MAC address:
Port MAC address 00
Aggregate group members: 4
ethernet1/21 ethernet1/22 ethernet1/23 ethernet1/24
Operation mode: layer3
Untagged sub-interface support: no
--------------------------------------------------------------------------------
Name: ae1, ID: 48
Operation mode: layer3
Interface management profile: N/A
Service configured: LACP
Zone: N/A, virtual system: vsys1
Adjust TCP MSS: no
--------------------------------------------------------------------------------

Hardware interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 817930387942
bytes transmitted 796893216522
packets received 1797090261
packets transmitted 1569474090
receive incoming errors 0
receive discarded 0
receive errors 64162355
packets dropped 0
--------------------------------------------------------------------------------

Logical interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 20819118
bytes transmitted 6130932
packets received 98682
packets transmitted 49443
receive errors 0
packets dropped 49325
packets dropped by flow state check 0

---------------------------------------

show interface ethernet1/21

--------------------------------------------------------------------------------
Name: ethernet1/21, ID: 36
Link status:
Runtime link speed/duplex/state: 10000/full/up
Configured link speed/duplex/state: auto/auto/auto
MAC address:
Port MAC address 00:1
Aggregate group : ae1
Operation mode: layer3
--------------------------------------------------------------------------------
Physical port counters read from MAC:
--------------------------------------------------------------------------------
rx-broadcast 2382266
rx-bytes 2428164611097
rx-multicast 1316048
rx-unicast 2213305474
tx-broadcast 1849
tx-bytes 1736956163089
tx-multicast 12326
tx-unicast 1653417852
--------------------------------------------------------------------------------

Hardware interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 0
bytes transmitted 0
packets received 0
packets transmitted 0
receive incoming errors 0
receive discarded 1
receive errors 0
packets dropped 0

why we can see these errores in ae interface?? why these erros are not shoed in ethernet interface in this ae???

Re: Several sessions outage/interface drops

Remo — Thu, 20 Jul 2017 18:49:12 GMT

When you create an aggregate ethernet, the counters move up from the physical interface to the logical one. The dropped packet counter normally almost always increases, unless you have an any-any-allow policy.

But the counter which you should be worried about is the receive error counter. Examples which result in such errors are broken cables, interfaces or transievers

Re: Several sessions outage/interface drops

BPry — Thu, 20 Jul 2017 19:09:58 GMT

@soporteseguridad,

@Remo already pointed out that you're actually looking at the wrong 'bad' figure in what you have listed. Are these SFP interfaces? I've seen interfaces have recieve issues showing on the PA side of things simply because of a dirty ferule.

More importantly your receive errors is what is alarming with the stats that you have posted and are far more likely to be your issue here. What type of switch do you have on the other end, do you see the same errors if you look on that end of the link?

Re: Several sessions outage/interface drops

TranceforLife — Fri, 21 Jul 2017 08:36:46 GMT

All,

I had a TAC case opened some time ago to get clarification for the "receive error" counters and below the feedback:

---------------------------------------------------------------------------------------------------------------------

I would like to share with you my findings from the log analysis. The receive error are logical errors although they are shown under Hardware interface errors. The possible events and packets could be the following incorrect length of VLAN tag, unexpected VLAN tag, unsupported L2 protocol, incorrect IP checksum, TCP/UDP packet checksum error, TCP/UDP port 0, Invalid TCP flag, etc. The following document has more details and explains how exactly this counters are working :

https://live.paloaltonetworks.com/t5/Learning-Articles/The-Difference-Between-Receive-Errors-for-Hardware-and-Logical/ta-p/59039

From my investigation I found that there is a high number of STP packets received which are not supported on the FW, dot1q tag errors, L4 checksum errors ( packets with TCP/UDP checksum not correct ). I attached the screenshots of our analysing tools in the file section of the case where you can check the high load with this type of traffic. In addition, the counter is accumulated since the last reboot of the device and therefore is the large number of errors. I hope this explanation will give you more details how exactly the counters are presented and what is the usual type of traffic triggering the counter to increase. If you have any additional questions in meantime, please feel free to contact us.

-------------------------------------------------------------------------------------------------------------------

I don't know if it is applicable to the ae interface (l think it should) but below the command that will get more inform:

amians@pxxxx(active)> show counter global filter severity drop

Global counters:
Elapsed time since last sampling: 1.248 seconds

name value rate severity category aspect description
--------------------------------------------------------------------------------
session_state_error 114323 0 drop session pktproc Session state error
session_dup_pkt_drop 16336698 20 drop session resource Duplicate packet: Applies only for multi-DP platform with hardware (Tiger) broadcasting pkt to all DPs
flow_rcv_err 16752983 24 drop flow parse Packets dropped: flow stage receive error
flow_rcv_dot1q_tag_err 26564200 16 drop flow parse Packets dropped: 802.1q tag not configured