topic Slow Performanced Based on Order of ACL Rules in General Topics

Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 15:03:14 GMT

We have several PAN 3020s at a client site with similar issues but for this, I’ll focus on a specific case. One pair in Active\Passive HA has 124 rules. We started noticing really slow RDP connect performance. (it would take 45 seconds to establish an RDP session to a target where the traffic was passed through the firewall). Out of the 124 rules, the rule which this RDP traffic matched on was around rule 100. If we moved that rule up earlier in the ACL to say, rule 5, the RDP session would only take 10 seconds or less to establish.

So initially, it is looking like the further down the ACL the rule is, the longer it takes the PAN to process that traffic. However, seeing that the 3020 supports up to 2500 policies and we only have 124, I wanted to check with you as it doesn’t seem right.

We are not doing any PBF here. Or App-ID override. Jumbo frames are enabled.

Re: Slow Performanced Based on Order of ACL Rules

HULK — Tue, 08 Jul 2014 15:15:55 GMT

Hello mackwage,

Would it be possible for you to take a packet capture for this traffic in both "receive" and "Transmit" stage. Then we can compare the processing time between receive and transmit in both conditions ( Rule at 5 and rule at 100).

What is the Data-plane utilization on this firewall including packet-rate, total active sessions, H/W nd S/W buffer, packet-descriptor, DP-CPU etc..?

Thanks

Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 15:23:49 GMT

Thanks for the reply! We can work on getting pcaps.

For the dataplane utilization, I am not sure where to get ALL of that info.

In the WebGUI, I can see the DP-CPU ranges between 2-6%. We ran show system statistics session and got.

The average throughput was 400-800k.

The sessions were ~900.

Re: Slow Performanced Based on Order of ACL Rules

hshah — Tue, 08 Jul 2014 15:27:14 GMT

Hi Mack,

Firewall is under utilized, hence its not over utilization issue. You are right, you can view MP/DP CPU on Dashboard.

Please provide us following output.

debug data-plane pool statistics

show system statistics session

show running resource-monitor

show session info

show counter global filter delta yes ( repeat the same command 5 times)

Regards,

Hardik shah

Re: Slow Performanced Based on Order of ACL Rules

HULK — Tue, 08 Jul 2014 15:32:27 GMT

PCAP along with the resource utilization would give us more insight about this issue.

Thanks

Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 15:58:30 GMT

debug data-plane pool statistics

Hardware Pools

[ 0] Packet Buffers : 11341/11468 0x8000000031000000

[ 1] Work Queue Entries : 229309/229376 0x8000000037ffe000

[ 2] Output Buffers : 1006/1024 0x8000000039bfe000

[ 3] DFA Result : 3999/4000 0x8000000039cfe000

[ 4] Timer Buffers : 4092/4096 0x800000003a0e6000

[ 5] PAN_FPA_LWM_POOL : 1024/1024 0x8000000000e9f200

[ 6] PAN_FPA_ZIP_POOL : 1023/1024 0x800000003a4e6000

[ 7] PAN_FPA_BLAST_POOL : 1024/1024 0x800000000ff00000

Software Pools

[ 0] software packet buffer 0 : 32767/32768 0x800000003a6e6680

[ 1] software packet buffer 1 : 32768/32768 0x800000003b706700

[ 2] software packet buffer 2 : 81920/81920 0x800000003d726780

[ 3] software packet buffer 3 : 20480/20480 0x8000000047776800

[ 4] software packet buffer 4 : 304/304 0x800000007018a880

[ 5] ZIP Results : 1024/1024 0x8000000084d4c0e0

[ 6] CTD Flow : 261635/262144 0x8000000084d66080

[ 7] CTD AV Block : 32/32 0x80000000a20bd340

[ 8] SML VM Fields : 524001/524288 0x80000000a20c5440

[ 9] SML VM Vchecks : 65536/65536 0x80000000a32c54c0

[10] Detector Threats : 196189/196608 0x80000000a3405540

[11] CTD DLP FLOW : 65536/65536 0x80000000a5da5608

[12] CTD DLP DATA : 4096/4096 0x80000000a65e5688

[13] CTD DECODE FILTER : 65536/65536 0x80000000a69e9710

[14] Regex Results : 8000/8000 0x80000000a6d4a088

[15] TIMER Chunk : 131072/131072 0x80000000aeca5ae0

[16] FPTCP segs : 32768/32768 0x80000000b0d25b60

[17] Proxy session : 7936/7936 0x80000000b0dc5be0

[18] SSL Handshake State : 7936/7936 0x80000000b1096860

[19] SSL State : 15872/15872 0x80000000b19564e0

[20] SSL Handshake MAC State : 17464/17464 0x80000000b25a0d60

[21] SSH Handshake State : 64/64 0x80000000b27d3ac0

[22] SSH State : 512/512 0x80000000b2841640

[23] TCP host connections : 15/16 0x80000000b297d020

show system statistics session

Device is up : 7 days 0 hour 39 mins 39 sec

Packet rate : 196/s

Throughput : 479 Kbps

Total active sessions : 818

Active TCP sessions : 503

Active UDP sessions : 311

Active ICMP sessions : 4

show running resource-monitor

Resource monitoring sampling data (per second):

CPU load sampling by group:

flow_lookup : 5%

flow_fastpath : 5%

flow_slowpath : 5%

flow_forwarding : 5%

flow_mgmt : 1%

flow_ctrl : 1%

nac_result : 5%

flow_np : 5%

dfa_result : 5%

module_internal : 5%

aho_result : 5%

zip_result : 5%

pktlog_forwarding : 6%

lwm : 0%

flow_host : 1%

show session info

--------------------------------------------------------------------------------

Number of sessions supported: 262142

Number of active sessions: 845

Number of active TCP sessions: 505

Number of active UDP sessions: 330

Number of active ICMP sessions: 10

Number of active BCAST sessions: 0

Number of active MCAST sessions: 0

Number of active predict sessions: 0

Session table utilization: 0%

Number of sessions created since bootup: 4899529

Packet rate: 296/s

Throughput: 955 kbps

New connection establish rate: 0 cps

--------------------------------------------------------------------------------

Session timeout

TCP default timeout: 3600 secs

TCP session timeout before SYN-ACK received: 5 secs

TCP session timeout before 3-way handshaking: 10 secs

TCP session timeout after FIN/RST: 30 secs

UDP default timeout: 30 secs

ICMP default timeout: 6 secs

other IP default timeout: 30 secs

Captive Portal session timeout: 30 secs

Session timeout in discard state:

TCP: 90 secs, UDP: 60 secs, other IP protocols: 60 secs

--------------------------------------------------------------------------------

Session accelerated aging: True

Accelerated aging threshold: 80% of utilization

Scaling factor: 2 X

--------------------------------------------------------------------------------

Session setup

TCP - reject non-SYN first packet: True

Hardware session offloading: True

IPv6 firewalling: True

--------------------------------------------------------------------------------

Application trickling scan parameters:

Timeout to determine application trickling: 10 secs

Resource utilization threshold to start scan: 80%

Scan scaling factor over regular aging: 8

--------------------------------------------------------------------------------

Session behavior when resource limit is reached: drop

--------------------------------------------------------------------------------

Pcap token bucket rate : 10485760

--------------------------------------------------------------------------------

Re: Slow Performanced Based on Order of ACL Rules

pulukas — Tue, 08 Jul 2014 15:59:59 GMT

I would take a look at the rules and organization. One of the things I've seen that can affect initial session setup time is the number of rules that are application only with the "any" port selection.

These rules should always be towards the bottom of the rule base.

The most specific rules with specified ports and applications should be at the top.

What happens with application and any rules is that PanOS essentially holds on that rule while the traffic is coming through and makes sure that the application is not a match before moving on to the next rule for checks. If there are a number of these "any" port rules each one does the check and passes on and the time delay can add up.

Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 16:00:24 GMT

show counter global filter delta yes ( repeat the same command 5 times)

Global counters:

Elapsed time since last sampling: 156.760 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------

pkt_recv 569924 3635 info packet pktproc Packets received

pkt_recv_zero 569190 3630 info packet pktproc Packets received from QoS 0

pkt_sent 559573 3569 info packet pktproc Packets transmitted

pkt_alloc 588942 3756 info packet resource Packets allocated

session_allocated 846 5 info session resource Sessions allocated

session_freed 874 5 info session resource Sessions freed

session_installed 843 5 info session resource Sessions installed

session_discard 3 0 info session resource Session set to discard by security policy check

flow_rcv_dot1q_tag_err 247 1 drop flow parse Packets dropped: 802.1q tag not configured

flow_no_interface 247 1 drop flow parse Packets dropped: invalid interface

flow_ipv6_disabled 471 3 drop flow parse Packets dropped: IPv6 disabled on interface

flow_policy_deny 176 1 drop flow session Session setup: denied by policy

flow_tcp_non_syn 52 0 info flow session Non-SYN TCP packets without session match

flow_tcp_non_syn_drop 52 0 drop flow session Packets dropped: non-SYN TCP without session match

flow_fwd_l3_bcast_drop 18 0 drop flow forward Packets dropped: unhandled IP broadcast

flow_fwd_l3_mcast_drop 376 2 drop flow forward Packets dropped: no route for IP multicast

flow_fwd_l3_noarp 58 0 drop flow forward Packets dropped: no ARP

flow_fwd_zonechange 31 0 drop flow forward Packets dropped: forwarded to different zone

flow_parse_unmatched_icmperr 77 0 info flow parse Packets dropped: Unmatched ICMP error message

flow_dos_rule_allow_under_rate 6 0 info flow dos Packets allowed: Rate within thresholds of DoS policy

flow_dos_rule_match 6 0 info flow dos Packets matched DoS policy

flow_dos_rule_nomatch 840 5 info flow dos Packets not matched DoS policy

flow_dos_ag_curr_sess_add_incr 6 0 info flow dos Incremented aggregate current session count on session create

flow_dos_ag_curr_sess_del_decr 5 0 info flow dos Decremented aggregate current session count on session delete

flow_dos_cl_curr_sess_add_incr 6 0 info flow dos Incremented classified current session count on session create

flow_dos_cl_curr_sess_del_decr 5 0 info flow dos Decremented classified current session count on session delete

flow_dos_ag_buckets_upd 314 2 info flow dos Updated aggregate buckets for aging

flow_action_close 2 0 drop flow pktproc TCP sessions closed via injecting RST

flow_arp_pkt_rcv 178 1 info flow arp ARP packets received

flow_arp_pkt_xmt 122 0 info flow arp ARP packets transmitted

flow_arp_pkt_replied 114 0 info flow arp ARP requests replied

flow_arp_rcv_gratuitous 19 0 info flow arp Gratuitous ARP packets received

flow_arp_resolve_xmt 26 0 info flow arp ARP resolution packets transmitted

flow_host_pkt_rcv 734 4 info flow mgmt Packets received from control plane

flow_host_pkt_xmt 1437 9 info flow mgmt Packets transmitted to control plane

flow_host_service_allow 95 0 info flow mgmt Device management session allowed

flow_health_monitor_rcv 673 4 info flow mgmt Health monitoring packet received

flow_health_monitor_xmt 673 4 info flow mgmt Health monitoring packet transmitted

appid_ident_by_icmp 201 1 info appid pktproc Application identified by icmp type

appid_ident_by_dport_first 156 0 info appid pktproc Application identified by L4 dport first

appid_ident_by_dport 12 0 info appid pktproc Application identified by L4 dport

appid_proc 443 2 info appid pktproc The number of packets processed by Application identification

appid_use_dfa_1 258 1 info appid pktproc The number of packets using the second DFA table

appid_unknown_fini_empty 109 0 info appid pktproc The number of unknown applications because of no data

nat_static_xlat 4 0 info nat resource The total number of static NAT translate called

nat_static_release 6 0 info nat resource The total number of static NAT release called

nat_dynamic_port_xlat 15 0 info nat resource The total number of dynamic_ip_port NAT translate called

nat_dynamic_port_release 15 0 info nat resource The total number of dynamic_ip_port NAT release called

dfa_dte_request_total 147094 938 info dfa offload The total number of dfa match using dte

dfa_hte_in_cache_lookup 146276 933 info dfa offload The total number of requests to an in cache HFA graph

tcp_case_2 209 1 info tcp pktproc tcp reassembly case 2

ctd_sml_exit_detector_i 27 0 info ctd pktproc The number of sessions with sml exit in detector i

appid_bypass_no_ctd 12 0 info appid pktproc appid bypass due to no ctd

ctd_err_bypass 27 0 info ctd pktproc ctd error bypass

ctd_run_pattern_match_failure 20 0 info ctd pktproc Run pattern match failure

ctd_do_pattern_match 20 0 info ctd pktproc do pattern match

ctd_sml_vm_run_impl_opcodeexit 27 0 info ctd pktproc SML VM opcode exit

[64;1H [K [7mlines 1-63 [27m [64;1H [64;1H [Kctd_sml_vm_run_impl_immed8000 662 4 info ctd pktproc SML VM immed8000

ctd_sml_vm_check_domain 159 1 info ctd pktproc sml vm check domain

ctd_sml_opcode_set_file_type 688 4 info ctd pktproc sml opcode set file type

ctd_filter_decode_failure_zip 34 0 error ctd pktproc Number of decode filter failure for zip

ctd_bloom_filter_nohit 162 1 info ctd pktproc The number of no match for virus bloom filter

ctd_bloom_filter_hit 9 0 info ctd pktproc The number of match for virus bloom filter

ctd_bloom_filter_pattern_notfound 9 0 info ctd pktproc The number of missing pattern match for virus bloom filter

aho_fpga 297239 1896 info aho resource The total requests to FPGA for AHO

aho_sw 97 0 info aho pktproc The total usage of software for AHO

ctd_appid_reassign 48318 308 info ctd pktproc appid was changed

ctd_url_block 1 0 info ctd pktproc sessions blocked by url filtering

ctd_pkt_slowpath 149601 954 info ctd pktproc Packets processed by slowpath

ha_msg_sent 290908 1855 info ha system HA: messages sent

ha_msg_recv 313 1 info ha system HA: messages received

ha_session_setup_msg_sent 712 4 info ha pktproc HA: session setup messages sent

ha_session_teardown_msg_sent 398 2 info ha pktproc HA: session teardown messages sent

ha_session_update_msg_sent 289539 1847 info ha pktproc HA: session update messages sent

ha_arp_update_msg_sent 102 0 info ha pktproc HA: ARP update messages sent

ha_ha2_monitor_msg_sent 313 1 info ha pktproc HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv 313 1 info ha pktproc HA: HA2 monitor message messages received

log_vulnerability_cnt 2066 13 info log system Number of vulnerability logs

log_fileext_cnt 18 0 info log system Number of file block logs

log_traffic_cnt 980 6 info log system Number of traffic logs

zip_process_total 104 0 info zip pktproc The total number of zip engine decompress process

zip_process_failure 34 0 info zip pktproc The number of failures for zip decompress process

pkt_nac_result 297239 1896 info packet resource Packets entered module nac stage result

pkt_flow_np 271951 1734 info packet resource Packets entered module flow stage np

pkt_flow_host 734 4 info packet resource Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 85

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 4.680 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------

pkt_recv 19543 4175 info packet pktproc Packets received

pkt_recv_zero 19522 4171 info packet pktproc Packets received from QoS 0

pkt_sent 17488 3736 info packet pktproc Packets transmitted

pkt_alloc 20124 4300 info packet resource Packets allocated

session_allocated 50 10 info session resource Sessions allocated

session_freed 18 3 info session resource Sessions freed

session_installed 50 10 info session resource Sessions installed

flow_rcv_dot1q_tag_err 6 1 drop flow parse Packets dropped: 802.1q tag not configured

flow_no_interface 6 1 drop flow parse Packets dropped: invalid interface

flow_ipv6_disabled 19 4 drop flow parse Packets dropped: IPv6 disabled on interface

flow_policy_deny 5 1 drop flow session Session setup: denied by policy

flow_fwd_l3_bcast_drop 2 0 drop flow forward Packets dropped: unhandled IP broadcast

flow_fwd_l3_mcast_drop 12 2 drop flow forward Packets dropped: no route for IP multicast

flow_fwd_l3_noarp 3 0 drop flow forward Packets dropped: no ARP

flow_fwd_zonechange 1 0 drop flow forward Packets dropped: forwarded to different zone

flow_parse_unmatched_icmperr 6 1 info flow parse Packets dropped: Unmatched ICMP error message

flow_dos_rule_nomatch 50 10 info flow dos Packets not matched DoS policy

flow_dos_ag_buckets_upd 9 1 info flow dos Updated aggregate buckets for aging

flow_arp_pkt_rcv 5 1 info flow arp ARP packets received

flow_arp_pkt_replied 1 0 info flow arp ARP requests replied

flow_host_pkt_rcv 21 4 info flow mgmt Packets received from control plane

flow_host_pkt_xmt 39 8 info flow mgmt Packets transmitted to control plane

flow_host_service_allow 4 0 info flow mgmt Device management session allowed

flow_health_monitor_rcv 18 3 info flow mgmt Health monitoring packet received

flow_health_monitor_xmt 18 3 info flow mgmt Health monitoring packet transmitted

appid_ident_by_icmp 12 2 info appid pktproc Application identified by icmp type

appid_ident_by_dport_first 1 0 info appid pktproc Application identified by L4 dport first

appid_proc 30 6 info appid pktproc The number of packets processed by Application identification

appid_use_dfa_1 13 2 info appid pktproc The number of packets using the second DFA table

appid_unknown_fini_empty 7 1 info appid pktproc The number of unknown applications because of no data

nat_static_xlat 2 0 info nat resource The total number of static NAT translate called

dfa_dte_request_total 5534 1182 info dfa offload The total number of dfa match using dte

dfa_hte_in_cache_lookup 5465 1167 info dfa offload The total number of requests to an in cache HFA graph

tcp_case_2 8 1 info tcp pktproc tcp reassembly case 2

ctd_sml_exit_detector_i 3 0 info ctd pktproc The number of sessions with sml exit in detector i

appid_bypass_no_ctd 3 0 info appid pktproc appid bypass due to no ctd

ctd_err_bypass 3 0 info ctd pktproc ctd error bypass

ctd_run_pattern_match_failure 1 0 info ctd pktproc Run pattern match failure

ctd_do_pattern_match 1 0 info ctd pktproc do pattern match

ctd_sml_vm_run_impl_opcodeexit 3 0 info ctd pktproc SML VM opcode exit

ctd_sml_vm_run_impl_immed8000 20 4 info ctd pktproc SML VM immed8000

ctd_sml_vm_check_domain 1 0 info ctd pktproc sml vm check domain

ctd_sml_opcode_set_file_type 21 4 info ctd pktproc sml opcode set file type

ctd_bloom_filter_nohit 1 0 info ctd pktproc The number of no match for virus bloom filter

aho_fpga 11017 2354 info aho resource The total requests to FPGA for AHO

aho_sw 14 2 info aho pktproc The total usage of software for AHO

ctd_appid_reassign 1871 399 info ctd pktproc appid was changed

ctd_pkt_slowpath 5573 1190 info ctd pktproc Packets processed by slowpath

ha_msg_sent 9091 1942 info ha system HA: messages sent

ha_msg_recv 10 2 info ha system HA: messages received

ha_session_setup_msg_sent 40 8 info ha pktproc HA: session setup messages sent

ha_session_teardown_msg_sent 8 1 info ha pktproc HA: session teardown messages sent

ha_session_update_msg_sent 9037 1930 info ha pktproc HA: session update messages sent

ha_arp_update_msg_sent 1 0 info ha pktproc HA: ARP update messages sent

ha_ha2_monitor_msg_sent 10 2 info ha pktproc HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv 10 2 info ha pktproc HA: HA2 monitor message messages received

log_vulnerability_cnt 48 10 info log system Number of vulnerability logs

[64;1H [K [7mlines 1-63 [27m [64;1H [64;1H [Klog_fileext_cnt 4 0 info log system Number of file block logs

log_traffic_cnt 23 4 info log system Number of traffic logs

pkt_nac_result 11017 2354 info packet resource Packets entered module nac stage result

pkt_flow_np 8506 1817 info packet resource Packets entered module flow stage np

pkt_flow_host 21 4 info packet resource Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 62

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 3.820 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------

pkt_recv 18988 4970 info packet pktproc Packets received

pkt_recv_zero 18967 4965 info packet pktproc Packets received from QoS 0

pkt_sent 16860 4413 info packet pktproc Packets transmitted

pkt_alloc 19541 5115 info packet resource Packets allocated

session_allocated 50 13 info session resource Sessions allocated

session_freed 18 4 info session resource Sessions freed

session_installed 50 13 info session resource Sessions installed

flow_rcv_dot1q_tag_err 7 1 drop flow parse Packets dropped: 802.1q tag not configured

flow_no_interface 7 1 drop flow parse Packets dropped: invalid interface

flow_ipv6_disabled 12 3 drop flow parse Packets dropped: IPv6 disabled on interface

flow_policy_deny 6 1 drop flow session Session setup: denied by policy

flow_tcp_non_syn 1 0 info flow session Non-SYN TCP packets without session match

flow_tcp_non_syn_drop 1 0 drop flow session Packets dropped: non-SYN TCP without session match

flow_fwd_l3_mcast_drop 9 2 drop flow forward Packets dropped: no route for IP multicast

flow_fwd_zonechange 1 0 drop flow forward Packets dropped: forwarded to different zone

flow_dos_rule_allow_under_rate 1 0 info flow dos Packets allowed: Rate within thresholds of DoS policy

flow_dos_rule_match 1 0 info flow dos Packets matched DoS policy

flow_dos_rule_nomatch 49 12 info flow dos Packets not matched DoS policy

flow_dos_ag_curr_sess_add_incr 1 0 info flow dos Incremented aggregate current session count on session create

flow_dos_cl_curr_sess_add_incr 1 0 info flow dos Incremented classified current session count on session create

flow_dos_ag_buckets_upd 8 2 info flow dos Updated aggregate buckets for aging

flow_arp_pkt_rcv 3 0 info flow arp ARP packets received

flow_arp_pkt_replied 3 0 info flow arp ARP requests replied

flow_host_pkt_rcv 21 5 info flow mgmt Packets received from control plane

flow_host_pkt_xmt 45 11 info flow mgmt Packets transmitted to control plane

flow_host_service_allow 3 0 info flow mgmt Device management session allowed

flow_health_monitor_rcv 19 4 info flow mgmt Health monitoring packet received

flow_health_monitor_xmt 19 4 info flow mgmt Health monitoring packet transmitted

appid_ident_by_icmp 8 2 info appid pktproc Application identified by icmp type

appid_ident_by_dport_first 16 4 info appid pktproc Application identified by L4 dport first

appid_proc 27 7 info appid pktproc The number of packets processed by Application identification

appid_use_dfa_1 12 3 info appid pktproc The number of packets using the second DFA table

appid_unknown_fini_empty 3 0 info appid pktproc The number of unknown applications because of no data

dfa_dte_request_total 5339 1397 info dfa offload The total number of dfa match using dte

dfa_hte_in_cache_lookup 5309 1389 info dfa offload The total number of requests to an in cache HFA graph

tcp_case_2 7 1 info tcp pktproc tcp reassembly case 2

ctd_sml_vm_run_impl_immed8000 19 4 info ctd pktproc SML VM immed8000

ctd_sml_vm_check_domain 16 4 info ctd pktproc sml vm check domain

ctd_sml_opcode_set_file_type 19 4 info ctd pktproc sml opcode set file type

ctd_filter_decode_failure_zip 7 1 error ctd pktproc Number of decode filter failure for zip

ctd_bloom_filter_nohit 16 4 info ctd pktproc The number of no match for virus bloom filter

aho_fpga 10766 2818 info aho resource The total requests to FPGA for AHO

aho_sw 2 0 info aho pktproc The total usage of software for AHO

ctd_appid_reassign 1814 474 info ctd pktproc appid was changed

ctd_pkt_slowpath 5435 1422 info ctd pktproc Packets processed by slowpath

ha_msg_sent 8751 2290 info ha system HA: messages sent

ha_msg_recv 8 2 info ha system HA: messages received

ha_session_setup_msg_sent 45 11 info ha pktproc HA: session setup messages sent

ha_session_teardown_msg_sent 8 2 info ha pktproc HA: session teardown messages sent

ha_session_update_msg_sent 8691 2275 info ha pktproc HA: session update messages sent

ha_arp_update_msg_sent 3 0 info ha pktproc HA: ARP update messages sent

ha_ha2_monitor_msg_sent 8 2 info ha pktproc HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv 8 2 info ha pktproc HA: HA2 monitor message messages received

log_url_req_cnt 2 0 info log system Number of url request logs

log_vulnerability_cnt 71 18 info log system Number of vulnerability logs

log_traffic_cnt 23 6 info log system Number of traffic logs

url_db_request 2 0 info url pktproc Number of URL database request

[64;1H [K [7mlines 1-63 [27m [64;1H [64;1H [Kurl_db_reply 2 0 info url pktproc Number of URL reply

zip_process_total 23 6 info zip pktproc The total number of zip engine decompress process

zip_process_failure 7 1 info zip pktproc The number of failures for zip decompress process

pkt_nac_result 10766 2818 info packet resource Packets entered module nac stage result

pkt_flow_np 8200 2146 info packet resource Packets entered module flow stage np

pkt_flow_host 21 5 info packet resource Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 63

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 4.722 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------

pkt_recv 18897 4001 info packet pktproc Packets received

pkt_recv_zero 18875 3997 info packet pktproc Packets received from QoS 0

pkt_sent 16528 3500 info packet pktproc Packets transmitted

pkt_alloc 19459 4120 info packet resource Packets allocated

session_allocated 33 6 info session resource Sessions allocated

session_freed 39 8 info session resource Sessions freed

session_installed 33 6 info session resource Sessions installed

session_discard 1 0 info session resource Session set to discard by security policy check

flow_rcv_dot1q_tag_err 9 1 drop flow parse Packets dropped: 802.1q tag not configured

flow_no_interface 9 1 drop flow parse Packets dropped: invalid interface

flow_ipv6_disabled 14 2 drop flow parse Packets dropped: IPv6 disabled on interface

flow_policy_deny 2 0 drop flow session Session setup: denied by policy

flow_fwd_l3_mcast_drop 13 2 drop flow forward Packets dropped: no route for IP multicast

flow_fwd_l3_noarp 3 0 drop flow forward Packets dropped: no ARP

flow_fwd_zonechange 1 0 drop flow forward Packets dropped: forwarded to different zone

flow_parse_unmatched_icmperr 6 1 info flow parse Packets dropped: Unmatched ICMP error message

flow_dos_rule_nomatch 33 6 info flow dos Packets not matched DoS policy

flow_dos_ag_buckets_upd 9 1 info flow dos Updated aggregate buckets for aging

flow_arp_pkt_rcv 4 0 info flow arp ARP packets received

flow_arp_pkt_xmt 1 0 info flow arp ARP packets transmitted

flow_arp_pkt_replied 2 0 info flow arp ARP requests replied

flow_arp_resolve_xmt 1 0 info flow arp ARP resolution packets transmitted

flow_host_pkt_rcv 22 4 info flow mgmt Packets received from control plane

flow_host_pkt_xmt 43 9 info flow mgmt Packets transmitted to control plane

flow_host_service_allow 1 0 info flow mgmt Device management session allowed

flow_health_monitor_rcv 21 4 info flow mgmt Health monitoring packet received

flow_health_monitor_xmt 21 4 info flow mgmt Health monitoring packet transmitted

appid_ident_by_icmp 9 1 info appid pktproc Application identified by icmp type

appid_ident_by_dport_first 5 1 info appid pktproc Application identified by L4 dport first

appid_proc 15 3 info appid pktproc The number of packets processed by Application identification

appid_use_dfa_1 13 2 info appid pktproc The number of packets using the second DFA table

appid_unknown_fini_empty 10 2 info appid pktproc The number of unknown applications because of no data

nat_dynamic_port_xlat 2 0 info nat resource The total number of dynamic_ip_port NAT translate called

dfa_dte_request_total 5384 1140 info dfa offload The total number of dfa match using dte

dfa_hte_in_cache_lookup 5361 1135 info dfa offload The total number of requests to an in cache HFA graph

tcp_case_2 3 0 info tcp pktproc tcp reassembly case 2

ctd_sml_exit_detector_i 1 0 info ctd pktproc The number of sessions with sml exit in detector i

appid_bypass_no_ctd 1 0 info appid pktproc appid bypass due to no ctd

ctd_err_bypass 1 0 info ctd pktproc ctd error bypass

ctd_run_pattern_match_failure 1 0 info ctd pktproc Run pattern match failure

ctd_do_pattern_match 1 0 info ctd pktproc do pattern match

ctd_sml_vm_run_impl_opcodeexit 1 0 info ctd pktproc SML VM opcode exit

ctd_sml_vm_run_impl_immed8000 20 4 info ctd pktproc SML VM immed8000

ctd_sml_vm_check_domain 3 0 info ctd pktproc sml vm check domain

ctd_sml_opcode_set_file_type 21 4 info ctd pktproc sml opcode set file type

ctd_bloom_filter_nohit 3 0 info ctd pktproc The number of no match for virus bloom filter

aho_fpga 10839 2295 info aho resource The total requests to FPGA for AHO

aho_sw 4 0 info aho pktproc The total usage of software for AHO

ctd_appid_reassign 1798 380 info ctd pktproc appid was changed

ctd_url_block 1 0 info ctd pktproc sessions blocked by url filtering

ctd_pkt_slowpath 5447 1153 info ctd pktproc Packets processed by slowpath

ha_msg_sent 8598 1820 info ha system HA: messages sent

ha_msg_recv 9 1 info ha system HA: messages received

ha_session_setup_msg_sent 24 5 info ha pktproc HA: session setup messages sent

ha_session_teardown_msg_sent 7 1 info ha pktproc HA: session teardown messages sent

ha_session_update_msg_sent 8560 1812 info ha pktproc HA: session update messages sent

ha_arp_update_msg_sent 3 0 info ha pktproc HA: ARP update messages sent

[64;1H [K [7mlines 1-63 [27m [64;1H [64;1H [Kha_ha2_monitor_msg_sent 9 1 info ha pktproc HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv 9 1 info ha pktproc HA: HA2 monitor message messages received

log_vulnerability_cnt 69 14 info log system Number of vulnerability logs

log_traffic_cnt 33 6 info log system Number of traffic logs

pkt_nac_result 10839 2295 info packet resource Packets entered module nac stage result

pkt_flow_np 8036 1701 info packet resource Packets entered module flow stage np

pkt_flow_host 22 4 info packet resource Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 64

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 3.18 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------

pkt_recv 1006 333 info packet pktproc Packets received

pkt_recv_zero 994 329 info packet pktproc Packets received from QoS 0

pkt_sent 1832 607 info packet pktproc Packets transmitted

pkt_alloc 1075 356 info packet resource Packets allocated

session_allocated 13 4 info session resource Sessions allocated

session_freed 15 4 info session resource Sessions freed

session_installed 13 4 info session resource Sessions installed

flow_rcv_dot1q_tag_err 4 1 drop flow parse Packets dropped: 802.1q tag not configured

flow_no_interface 4 1 drop flow parse Packets dropped: invalid interface

flow_ipv6_disabled 6 1 drop flow parse Packets dropped: IPv6 disabled on interface

flow_policy_deny 3 0 drop flow session Session setup: denied by policy

flow_fwd_l3_mcast_drop 9 2 drop flow forward Packets dropped: no route for IP multicast

flow_fwd_l3_noarp 1 0 drop flow forward Packets dropped: no ARP

flow_parse_unmatched_icmperr 1 0 info flow parse Packets dropped: Unmatched ICMP error message

flow_dos_rule_allow_under_rate 1 0 info flow dos Packets allowed: Rate within thresholds of DoS policy

flow_dos_rule_match 1 0 info flow dos Packets matched DoS policy

flow_dos_rule_nomatch 12 3 info flow dos Packets not matched DoS policy

flow_dos_ag_curr_sess_add_incr 1 0 info flow dos Incremented aggregate current session count on session create

flow_dos_cl_curr_sess_add_incr 1 0 info flow dos Incremented classified current session count on session create

flow_dos_ag_buckets_upd 6 1 info flow dos Updated aggregate buckets for aging

flow_arp_pkt_rcv 3 0 info flow arp ARP packets received

flow_arp_pkt_replied 1 0 info flow arp ARP requests replied

flow_arp_rcv_gratuitous 1 0 info flow arp Gratuitous ARP packets received

flow_host_pkt_rcv 12 3 info flow mgmt Packets received from control plane

flow_host_pkt_xmt 19 6 info flow mgmt Packets transmitted to control plane

flow_host_service_allow 2 0 info flow mgmt Device management session allowed

flow_health_monitor_rcv 11 3 info flow mgmt Health monitoring packet received

flow_health_monitor_xmt 11 3 info flow mgmt Health monitoring packet transmitted

appid_ident_by_icmp 4 1 info appid pktproc Application identified by icmp type

appid_proc 9 2 info appid pktproc The number of packets processed by Application identification

appid_use_dfa_1 7 2 info appid pktproc The number of packets using the second DFA table

dfa_dte_request_total 106 35 info dfa offload The total number of dfa match using dte

dfa_hte_in_cache_lookup 99 32 info dfa offload The total number of requests to an in cache HFA graph

ctd_sml_vm_run_impl_immed8000 12 3 info ctd pktproc SML VM immed8000

ctd_sml_opcode_set_file_type 15 4 info ctd pktproc sml opcode set file type

aho_fpga 85 28 info aho resource The total requests to FPGA for AHO

ctd_pkt_slowpath 102 33 info ctd pktproc Packets processed by slowpath

ha_msg_sent 984 326 info ha system HA: messages sent

ha_msg_recv 6 1 info ha system HA: messages received

ha_session_setup_msg_sent 11 3 info ha pktproc HA: session setup messages sent

ha_session_teardown_msg_sent 7 2 info ha pktproc HA: session teardown messages sent

ha_session_update_msg_sent 962 318 info ha pktproc HA: session update messages sent

ha_arp_update_msg_sent 1 0 info ha pktproc HA: ARP update messages sent

ha_ha2_monitor_msg_sent 6 1 info ha pktproc HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv 6 1 info ha pktproc HA: HA2 monitor message messages received

log_traffic_cnt 21 6 info log system Number of traffic logs

pkt_nac_result 85 28 info packet resource Packets entered module nac stage result

pkt_flow_np 909 301 info packet resource Packets entered module flow stage np

pkt_flow_host 12 3 info packet resource Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 49

--------------------------------------------------------------------------------

Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 16:03:47 GMT

Thanks for your input!

In this firewall, we do not have any rules which have an application specified along with service ANY. We do however have several rules which list an application with service "application-default". Is it faster processing wise to explicitly list the port as oppose to app default?

Re: Slow Performanced Based on Order of ACL Rules

hshah — Tue, 08 Jul 2014 16:03:47 GMT

Hello Mackwage,

Output looks good, no issue. Would it be possible to provide packet captures.

Regards,

Hardik Shah

Re: Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 16:06:15 GMT

Will get those to you shortly. One question on the buffer output...

Hardware Pools

[ 0] Packet Buffers : 11341/11468 0x8000000031000000

[ 1] Work Queue Entries : 229309/229376 0x8000000037ffe000

[ 2] Output Buffers : 1006/1024 0x8000000039bfe000

[ 3] DFA Result : 3999/4000 0x8000000039cfe000

[ 4] Timer Buffers : 4092/4096 0x800000003a0e6000

[ 5] PAN_FPA_LWM_POOL : 1024/1024 0x8000000000e9f200

[ 6] PAN_FPA_ZIP_POOL : 1023/1024 0x800000003a4e6000

[ 7] PAN_FPA_BLAST_POOL : 1024/1024 0x800000000ff00000

Software Pools

[ 0] software packet buffer 0 : 32767/32768 0x800000003a6e6680

[ 1] software packet buffer 1 : 32768/32768 0x800000003b706700

[ 2] software packet buffer 2 : 81920/81920 0x800000003d726780

[ 3] software packet buffer 3 : 20480/20480 0x8000000047776800

The buffers say for example 32767/32768. Does this imply the buffer is almost full? Or is it specifying it is empty?

Re: Re: Slow Performanced Based on Order of ACL Rules

hshah — Tue, 08 Jul 2014 16:07:49 GMT

Buffers are empty, buffers are not utilized, buffers are under utilized.

Re: Slow Performanced Based on Order of ACL Rules

sspringer — Tue, 08 Jul 2014 16:18:45 GMT

There are some drop counters..

flow_policy_deny 6 1 drop flow session Session setup: denied by policy

flow_tcp_non_syn 1 0 info flow session Non-SYN TCP packets without session match

flow_tcp_non_syn_drop 1 0 drop flow session Packets dropped: non-SYN TCP without session match

flow_fwd_l3_mcast_drop 9 2 drop flow forward Packets dropped: no route for IP multicast

flow_fwd_zonechange 1 0 drop flow forward Packets dropped: forwarded to different zone

Could you configure a packet-filter based on the source and destination used for testing and collect new counters?

> debug dataplane packet-diag set filter match source x.x.x.x destination y.y.y.y

> debug dataplane packet-diag set filter on

Then collect counters with packet-filter

> show counter global name filter packet-filter yes delta yes

*Note* the filter will only match new sessions, not sessions already present in session table

Re: Re: Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 16:23:08 GMT

I have the pcaps but I am not seeing a forum option to upload attachments. :smileylaugh:

Re: Slow Performanced Based on Order of ACL Rules

pulukas — Tue, 08 Jul 2014 16:25:47 GMT

mackwage wrote:

Thanks for your input!

I don't have any direct experience with that situation. I would assume that it would only affect your target traffic if one of the default application ports overlapped with your delayed traffic.

Re: Re: Re: Slow Performanced Based on Order of ACL Rules

hshah — Tue, 08 Jul 2014 16:26:22 GMT

Click on User Advanced Editor, it will give options to add files.

Re: Re: Re: Re: Slow Performanced Based on Order of ACL Rules

SDorsey — Tue, 08 Jul 2014 16:47:15 GMT

Note that even on rule 5, establishing the RDP session takes 18 seconds (while going through PAN) which still seems like a long time compared to other environments.

Re: Slow Performanced Based on Order of ACL Rules

cpainchaud — Mon, 14 Jul 2014 12:09:25 GMT

There is no reason to see such a delay. You should check that no rule is denying connections from client to server and server to anywhere else (a bit like SSH when DNS is filtered on server side).

If you still experience that problem, you should open a TAC case.