topic Re: Detailed troubleshooting of drop counters in General Topics

Detailed troubleshooting of drop counters

arvesynd — Mon, 28 Oct 2013 09:01:37 GMT

Hi,

I am having some issues with odd packet drops, and "show counter global filter severity drop" shows a lot of packets being dropped due to "Packets dropped: 802.1q tag not configured/Packets dropped: invalid interface" (same amount of packets dropped on both, so I assume these are related).

Are there any way of getting a log of what has been happening for flow_rcv_dot1q_tag_err and flow_no_interface?

Just knowing that the firewall is dropping packets doesn't help a lot when I'm unable to get any further information about the traffic.

Re: Detailed troubleshooting of drop counters

VinceM — Mon, 28 Oct 2013 11:23:39 GMT

Hi,

Please look on:

https://live.paloaltonetworks.com/docs/DOC-3938

Does it make sense for your archie ?

Hope help.

Re: Detailed troubleshooting of drop counters

arvesynd — Mon, 28 Oct 2013 13:45:12 GMT

Hi Vince,

Yes, it does make sense, as all the Cisco devices in our network are running pvst, but the few of our remaining Dell PowerConnect switches seems to be unable to handle pvst.

We are troubleshooting some major issues with the network, and I would like to see the traffic that the firewall is dropping to see if this is related.

Since the firewall is logging that the traffic is being dropped, it should also log the actual traffic?

Re: Detailed troubleshooting of drop counters

mbutt — Mon, 28 Oct 2013 15:33:07 GMT

Hi you can do the captures on the firewall to see what traffic is being dropped. However this should be used only for the troubleshooting purposes in short burst. If you leave it running the pcap exceeds certain limit a new pcap file will be generated. But again this is CPU intensive and should be used only for troubleshooting.

In captures there are 4 different stages recieve, transmit, Drop and firewall. The drop pcap should show you what is being dropped by the firewall. Here are some additional tips as well.

1. Need to setup the filters for the traffic we are interested in. To do this, execute the following steps:

Navigate to Monitor--Packet Capture

Click 'Manage Filters'

Set Filter ID 1 to be the source IP and destination IP of traffic you feel is affected ( leave all other fields blank )

Set Filter ID 2 to be the exact inverse of what you did in step 3 (destination IP in source field, Source IP in destination field)

2. Setup up the captures

Create and name the file stage for a packet capture on all the stages (receive, transmit, firewall and drop)

3. Enable filters and captures

debug dataplane packet-diag set filter on

debug dataplane packet-diag set capture on

4. open 2 CLI windows

on 1 run the following command to look at the counter ( make sure it run this command once before running the traffic)

show counter global filter packet-filter yes delta yes

on the 2nd window run the following command to look at he sessions

show session all filter source <ip address> destination <ip address>

After your test has been done stop all the captures and filters and see if global counter show you anything why it is dropping the traffic or if you have getting pcap with drop stage.

This will help you narrow down the issue.

Let us know if this helps you resolve the issue.

Thanks

Numan

Re: Detailed troubleshooting of drop counters

arvesynd — Tue, 29 Oct 2013 06:58:21 GMT

Hi mbutt,

Thanks for your reply. I am aware of the packet capture feature, and I have used it to some extent, but I still can't help but feel that this is a huge detour for a problem that the firewall has detected, and done something about.

When troubleshooting PANOS compared to other manufacturers, I feel like the PAN firewall is hiding a lot of what it's actually doing from me, and instead focusing on presenting the traffic that runs smoothly with nice graphs.

Re: Detailed troubleshooting of drop counters

ldavie — Wed, 13 Nov 2013 23:53:58 GMT

I completely agree with you. We were troubleshooting issues
with packets being dropped and found it very hard to impossible to actually
find out why a packet is dropped. If the firewall drops a packet I should be
able to see a report or enter a CLI command to find out why and how many. I’m
actually filing a feature request for this. It just seems so basic, I can’t believe
it is so difficult.

Re: Detailed troubleshooting of drop counters

HULK — Thu, 14 Nov 2013 01:53:15 GMT

Hello,

You can follow this document and get a details information about any drop counter.

How to Enable Logging for Global Counters?

To enable logging for global counters, run the following command via CLI:

debug dataplane packet-diag set log counter <counter name Example:"flow_rcv_dot1q_tag_err and flow_no_interface">

You will be able to see the same logs under Monitor>> System logs.

To disable logging, run:

debug dataplane packet-diag clear log counter <counter_name>

Thanks