Hello everyone,
I wanted to share some knowledge I've gained about investigating common Layer 1 to Layer 4 issues, such as MTU mismatches and DoS attacks, using key Palo Alto Networks firewall features like Global Counters, Flow Debug, and packet captures.
The first steps in troubleshooting these issues are always to check your routing, run a policy trace, and review the global counters. Once you have a clearer picture, you can move on to more advanced tools like packet capture and flow debug. The following links provide a great starting point for those tools:
1. MTU Investigation
One of the first issues to investigate is an MTU mismatch. This can happen, for example, with a Palo Alto GRE or IPSEC tunnel to another system where the MTU between the two systems is too small, or the MTU of the packets traversing the firewall is too big and the "DF" (Don't Fragment) bit is set.
To add filters you can use the cli as shown in articles Tips & Tricks: Flow Basic Debugging and Tips & Tricks: App-ID Debugging | Palo Alto Networks
To demonstrate the most basic test, you can set the MTU on a test client-facing interface to 1400 and run a ping command. After the test, you can use the commands show counter global filter delta yes and show counter global filter delta yes severity drop.
delta yes shows you new statistics that were not seen in the previous command execution.
severity drop allows you to focus specifically on dropped traffic.
When you run the command without the severity drop filter, you can see other interesting things, such as how an application was identified or if there is unicast or multicast routing. I have described application identification in How to Write Palo Alto Networks Custom Vulnerability and Application Signatures with Examples | Palo....
show counter global filter delta yes
Global counters:
Elapsed time since last sampling: 45.110 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
pkt_recv 418 9 info packet pktproc Packets received
pkt_runtc_np 418 9 info packet pktproc runtc flow_np
pkt_sent 7 0 info packet pktproc Packets transmitted
pkt_sent_host 3 0 info packet pktproc Packets successfully transmitted to host interface
pkt_stp_rcv 23 0 info packet pktproc STP BPDU packets received
session_allocated 7 0 info session resource Sessions allocated
session_freed 8 0 info session resource Sessions freed
session_installed 7 0 info session resource Sessions installed
flow_rcv_dot1q_tag_err 309 6 drop flow parse Packets dropped: 802.1q tag not configured
flow_no_interface 309 6 drop flow parse Packets dropped: invalid interface
flow_fwd_l3_bcast_drop 1 0 drop flow forward Packets dropped: unhandled IP broadcast
flow_fwd_l3_mcast_drop 44 0 drop flow forward Packets dropped: no route for IP multicast
flow_icmp_err_not_passing_thru 1 0 drop flow ipsec ICMP error packet dropped: no IP configured on the interface
flow_fwd_ip_df_drop 1 0 drop flow forward Packets dropped: exceeded MTU but DF bit present
flow_ip6_mcast_off 27 0 info flow pktproc Packets received: IPv6 multicast pkts with flow off
flow_bcast_pkt_rcv 1 0 info flow parse IP broadcast pkt received
flow_arp_pkt_rcv 32 0 info flow arp ARP packets received
flow_arp_pkt_replied 1 0 info flow arp ARP requests replied
flow_arp_rcv_gratuitous 1 0 info flow arp Gratuitous ARP packets received
flow_host_pkt_xmt 3 0 info flow mgmt Packets transmitted to control plane
flow_ip_cksm_sw_validation 7 0 info flow pktproc Packets for which IP checksum validation was done in software
appid_ident_by_icmp 7 0 info appid pktproc Application identified by icmp type
dfa_sw 7 0 info dfa pktproc The total number of dfa match using software
ctd_pscan_sw 7 0 info ctd pktproc The total usage of software for pscan
ctd_process 7 0 info ctd pktproc session processed by ctd
ctd_pkt_slowpath 7 0 info ctd pktproc Packets processed by slowpath
log_traffic_cnt 3 0 info log system Number of traffic logs
log_suppress 8 0 info log system Logs suppressed by log suppression
--------------------------------------------------------------------------------
Total counters shown: 28
--------------------------------------------------------------------------------
Note: I once saw an issue where the intrazone-default rule was set to deny. In the global counters, the drop reason was seen as "Session setup: denied by policy," which masked the true MTU drop reason. The workaround was to create a specific rule for the client-facing zone that allows the traffic under investigation. The true MTU counter was only visible after the security rule was fixed.
The counter "Packets dropped: exceeded MTU but DF bit present" is seen in the second execution of the command after the issue with the security zone rule is fixed.
Extra links:
2. DOS investigation
DoS protection on Palo Alto Networks firewalls is based on Zone Protection profiles or DoS Protection security rules as shown in Zone Protection and DoS Protection.. The global counters are a great way to see traffic dropped by DoS policies. This is useful because if you correlate these drops with CPU and memory commands, you may discover that a spike in resource usage was not caused by a memory leak, but by a DoS attack.
A useful command for this is show counter global filter aspect dos delta yes. You can easily test this with a DoS policy that drops ICMP and just run a ping. I've had issues triggering this with a Zone Protection profile on a test VM, but a DoS policy worked fine.
The show counter global filter aspect dos delta yes command is useful even if you can't limit the global counters by source and destination, though limiting to a specific source and destination will still work. as seen below.
Also, show counter global filter category ssl delta yes is useful, as Layer 7 DoS attacks inside SSL (usually HTTPS web traffic) can also cause CPU and memory spikes.
If the issue turns out to be a resource leak due to a bug, an upgrade can be planned. As a workaround before that, you can schedule a process restart at night for the process causing the issue, as I've shown in my other article, Automating the Palo Alto NGFW's Process/Deamon Restarts.
3. Strata Cloud Manager/AIOps
Palo Alto's Strata Cloud Manager, which can manage Palo Alto Networks NGFWs and Prisma Access from a single web interface, now includes the AIOps feature. AIOps ingests global counters and can generate alerts for things like MTU traffic issues or when a DoS rule is triggered.
There is a free version and a premium one, which are described in the link Free and Premium Features.
With AIOps, you can even configure email or SMS notifications for these alerts!
Extra links:
