01-29-2021 08:34 AM - edited 01-29-2021 08:48 AM
We have below settings for our untrust zone protection. We don't see a high CPS rate but we still see packets getting dropped, and has now started effecting us. Any guidance would be helpful.
PANO9.0.11/5250
01-31-2021 03:30 PM
during a time you're seeing these drops, see if you are picking up syn cookie error counters:
> show counter global delta yes filter aspect dos
and look for any of these counters:
flow_dos_syncookie_ack_err info TCP SYN cookies: Invalid ACKs received, aggregate profile/zone
flow_dos_syncookie_blk_dur drop Packets dropped: Flagged for blocking and under block duration for ag
flow_dos_syncookie_max drop Packet dropped: SYN cookies maximum threshold reached, aggregate prof
flow_dos_syncookie_not_tcp_syn drop TCP SYN cookies: TCP SYN cookie not SYN
flow_dos_syncookie_not_tcp_syn_ack drop TCP SYN cookies: TCP SYN cookie not SYN-ACK
02-01-2021 08:06 AM
@reaper Its difficult to so when it happens as it lasts only few minutes, and not predictable when it would happen.
But because it had been effecting users I had to increase the activation to rate to 4000.
Below is the output though and i only see 1 flow that matches what you said to look for. flow_dos_syncookie_ack_er
show counter global filter delta yes aspect dos
Global counters:
Elapsed time since last sampling: 5.684 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_dos_syncookie_cookie_sent 2170 381 info flow dos TCP SYN cookies: cookies sent, aggregate profile/zone
flow_dos_syncookie_ack_rcv 2332 410 info flow dos TCP SYN cookies: ACKs to cookies received, aggregate profile/zone
flow_dos_syncookie_ack_err 81 14 info flow dos TCP SYN cookies: Invalid ACKs received, aggregate profile/zone
flow_dos_syncookie_svr_ack_rcv 2076 365 info flow dos TCP SYN cookies: Server ACKs received, aggregate profile/zone
flow_dos_syncookie_not_tcp_syn 304 53 drop flow dos TCP SYN cookies: TCP SYN cookie not SYN
flow_dos_pf_ipspoof 1 0 drop flow dos Packets dropped: Zone protection option 'discard-ip-spoof'
flow_dos_pf_ipfrag 24 4 drop flow dos Packets dropped: Zone protection option 'discard-ip-frag'
flow_dos_pf_ping0 10 1 drop flow dos Packets dropped: Zone protection option 'discard-icmp-ping-zero-id'
flow_dos_pf_icmplpkt 1 0 drop flow dos Packets dropped: Zone protection option 'discard-icmp-large-packet'
flow_dos_pm_tcptimestamp 65 11 info flow dos Packets modified: Zone protection option 'remove-tcp-timestamp'
flow_dos_rule_allow_under_rate 24 4 info flow dos Packets allowed: Rate within thresholds of DoS policy
flow_dos_rule_match 24 4 info flow dos Packets matched DoS policy
flow_dos_rule_nomatch 3800 668 info flow dos Packets not matched DoS policy
flow_dos_cl_curr_sess_add_incr 14 2 info flow dos Incremented classified current session count on session create
flow_dos_cl_curr_sess_del_decr 30 5 info flow dos Decremented classified current session count on session delete
flow_dos_ag_buckets_upd 11 1 info flow dos Updated aggregate buckets for aging
--------------------------------------------------------------------------------
Total counters shown: 16
--------------------------------------------------------------------------------02-01-2021 08:19 AM
Check to see if you have a lot of non_syn_tcp
show counter global filter aspect session category flow severity drop delta yes
02-01-2021 01:47 PM
If you are having trouble checking counters when experiencing the issue, I would simply script it and log the results on a regular basis. That way, when you experience the issue again you simply have to go back and look at the logged data to see if you can capture the counters increasing at a high rate.
02-02-2021 07:43 AM
@BPry Would you have a sample script for just that.
@reaper This seems quite bursty
show counter global filter aspect session category flow severity drop delta yes
Global counters:
Elapsed time since last sampling: 8.763 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_policy_deny 3575 407 drop flow session Session setup: denied by policy
flow_tcp_non_syn_drop 19507 2226 drop flow session Packets dropped: non-SYN TCP without session match
--------------------------------------------------------------------------------
Total counters shown: 2
--------------------------------------------------------------------------------
Global counters:
Elapsed time since last sampling: 1.548 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_policy_deny 636 410 drop flow session Session setup: denied by policy
flow_tcp_non_syn_drop 3530 2280 drop flow session Packets dropped: non-SYN TCP without session match
--------------------------------------------------------------------------------
Total counters shown: 2
--------------------------------------------------------------------------------
Also from what i have understood these are global counters, and they are just telling something is happening but don't tell where. Could the counters be not increasing because of something internal to our network for these counters.
This is what I seen with show zone-protection
show zone-protection zone EXTERNAL
------------------------------------------------------------------------------------------
Number of zones with protection profile: 1
------------------------------------------------------------------------------------------
Zone EXTERNAL, vsys vsys1, profile EXTERNAL
------------------------------------------------------------------------------------------
tcp-syn SYN cookie enabled: yes
DP alarm rate: 7000 cps, activate rate: 4500 cps, maximal rate: 35000 cps
current: 84 packets
dropped: 0 packets
not TCP SYN: 196153882 packets
not TCP SYN ACK: 25849 packets
------------------------------------------------------------------------------------------
udp RED enabled: yes
DP alarm rate: 5000 cps, activate rate: 15000 cps, maximal rate: 30000 cps
current: 2 packets
dropped: 0 packets
------------------------------------------------------------------------------------------
icmp RED enabled: yes
DP alarm rate: 5000 cps, activate rate: 15000 cps, maximal rate: 30000 cps
current: 0 packets
dropped: 0 packets
------------------------------------------------------------------------------------------
other-ip RED enabled: yes
DP alarm rate: 5000 cps, activate rate: 15000 cps, maximal rate: 30000 cps
current: 0 packets
dropped: 0 packets
------------------------------------------------------------------------------------------
icmpv6 RED enabled: yes
DP alarm rate: 5000 cps, activate rate: 15000 cps, maximal rate: 30000 cps
current: 0 packets
dropped: 0 packets
------------------------------------------------------------------------------------------
IPv(4/6) Filter:
discard-ip-spoof: enabled: yes, packet dropped: 121
discard-ip-frag: enabled: yes, packet dropped: 3146
tcp-reject-non-syn: enabled: yes, (global), packet dropped: 108198
tcp-timestamp: enabled: yes, packets modified: 4370
discard-tcp-syn-with-data: enabled: yes, packet dropped: 0
discard-tcp-synack-with-data: enabled: yes, packet dropped: 0
strip-tcp-fast-open-and-data: enabled: yes, packet stripped: 9
IPv4 packet filter:
discard-icmp-ping-zero-id: enabled: yes, packet dropped: 30
discard-icmp-frag: enabled: yes, packet dropped: 0
discard-icmp-large-packet: enabled: yes, packet dropped: 0
02-02-2021 03:55 PM
I have made this powershell script to get zone information . There might be easy ways, but this is what I got to work and log the output
#Loop endlessly with 5 second delay w
while($true){
$Login = Invoke-restmethod -Uri 'https://pan/api/?type=keygen&user=xml&password=P@ssword1'
$LoginXML = Select-Xml -Xml $Login -XPath "/response[@status]/result/key/text()" | Select-Object -ExpandProperty Node
$Key = $LoginXML.Value
$XMLURI = 'https://pan/api/?type=op&cmd=<show><zone-protection><zone>EXTERNAL</zone></zone-protection></show>&key=' + $Key
$XMLURI
$Request = Invoke-restmethod -Uri $XMLURI
$Parse0 = Select-Xml -Xml $Request -XPath "/response[@status]/result/entry/entries/entry" | Select-Object -ExpandProperty Node
$Parse1 = Select-Xml -Xml $Request -XPath "/response[@status]/result/entry/entries/entry/other-ip" | Select-Object -ExpandProperty Node
$Parse2 = Select-Xml -Xml $Request -XPath "/response[@status]/result/entry/entries/entry/tcp-syn" | Select-Object -ExpandProperty Node
$Parse3 = Select-Xml -Xml $Request -XPath "/response[@status]/result/entry/entries/entry/udp" | Select-Object -ExpandProperty Node
$Parse4 = Select-Xml -Xml $Request -XPath "/response[@status]/result/entry/entries/entry/icmp" | Select-Object -ExpandProperty Node
$Parse5 = Select-Xml -Xml $Request -XPath "/response[@status]/result/entry/entries/entry/icmpv6" | Select-Object -ExpandProperty Node
$Time = (Get-Date).ToString(‘yyyy-MM-dd HH:mm:ss’)
$Parse0 | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time -Passthru | Export-Csv -Path "d:\palo\Entry.csv" -Append
$Parse1 | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time -Passthru | Export-Csv -Path "d:\palo\OTHER-IP.csv" -Append
$Parse2 | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time -Passthru | Export-Csv -Path "d:\palo\TCP-SYN.csv" -Append
$Parse3 | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time -Passthru | Export-Csv -Path "d:\palo\UDP.CSV" -Append
$Parse4 | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time -Passthru | Export-Csv -Path "d:\palo\ICMP.csv" -Append
$Parse5 | Add-Member -MemberType NoteProperty -Name Timestamp -Value $Time -Passthru | Export-Csv -Path "d:\palo\ICMP6.csv" -Append
Start-Sleep -s 5
}
02-02-2021 04:40 PM
No, the potential to cause a catastrophic load on your device is too high IMHO
If you're at the end of your rope, open a case with support so they can assist during a live session
02-02-2021 08:10 PM
I've never had a need to programmatically pull global counters on an automated basis outside of specific troubleshooting periods where I was already in maintenance windows. I'm not sure if it would put enough load on the firewall to cause any issues; I don't recall it placing an abnormal load on the system in the past, but I was also in maintenance windows and wouldn't really have been looking for it.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
