Slow Performanced Based on Order of ACL Rules

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Slow Performanced Based on Order of ACL Rules

L4 Transporter

We have several PAN 3020s at a client site with similar issues but for this, I’ll focus on a specific case. One pair in Active\Passive HA has 124 rules. We started noticing really slow RDP connect performance. (it would take 45 seconds to establish an RDP session to a target where the traffic was passed through the firewall). Out of the 124 rules, the rule which this RDP traffic matched on was around rule 100. If we moved that rule up earlier in the ACL to say, rule 5, the RDP session would only take 10 seconds or less to establish.

So initially, it is looking like the further down the ACL the rule is, the longer it takes the PAN to process that traffic. However, seeing that the 3020 supports up to 2500 policies and we only have 124, I wanted to check with you as it doesn’t seem right.

We are not doing any PBF here. Or App-ID override. Jumbo frames are enabled.

17 REPLIES 17

L7 Applicator

Hello mackwage,

Would it be possible for you to take a packet capture for this traffic in both "receive" and "Transmit" stage. Then we can compare the processing time between receive and transmit in both conditions ( Rule at 5 and rule at 100).

What is the Data-plane utilization on this firewall including packet-rate, total active sessions, H/W nd S/W buffer, packet-descriptor, DP-CPU etc..?

Thanks

Thanks for the reply! We can work on getting pcaps.

For the dataplane utilization, I am not sure where to get ALL of that info.

In the WebGUI, I can see the DP-CPU ranges between 2-6%. We ran show system statistics session and got.

The average throughput was 400-800k.

The sessions were ~900.

Hi Mack,

Firewall is under utilized, hence its not over utilization issue. You are right, you can view MP/DP CPU on Dashboard.

Please provide us following output.

debug data-plane pool statistics

show system statistics session

show running resource-monitor

show session info

show counter global filter delta yes  ( repeat the same command 5 times)

Regards,

Hardik shah

PCAP along with the resource utilization would give us more insight about this issue.

Thanks

debug data-plane pool statistics

Hardware Pools

[ 0] Packet Buffers :    11341/11468    0x8000000031000000

[ 1] Work Queue Entries        : 229309/229376   0x8000000037ffe000

[ 2] Output Buffers :     1006/1024     0x8000000039bfe000

[ 3] DFA Result :     3999/4000     0x8000000039cfe000

[ 4] Timer Buffers :     4092/4096     0x800000003a0e6000

[ 5] PAN_FPA_LWM_POOL :     1024/1024     0x8000000000e9f200

[ 6] PAN_FPA_ZIP_POOL :     1023/1024     0x800000003a4e6000

[ 7] PAN_FPA_BLAST_POOL :     1024/1024     0x800000000ff00000

Software Pools

[ 0] software packet buffer 0  : 32767/32768    0x800000003a6e6680

[ 1] software packet buffer 1  : 32768/32768    0x800000003b706700

[ 2] software packet buffer 2  : 81920/81920    0x800000003d726780

[ 3] software packet buffer 3  : 20480/20480    0x8000000047776800

[ 4] software packet buffer 4  :      304/304 0x800000007018a880

[ 5] ZIP Results :     1024/1024     0x8000000084d4c0e0

[ 6] CTD Flow :   261635/262144   0x8000000084d66080

[ 7] CTD AV Block : 32/32       0x80000000a20bd340

[ 8] SML VM Fields :   524001/524288   0x80000000a20c5440

[ 9] SML VM Vchecks :    65536/65536    0x80000000a32c54c0

[10] Detector Threats :   196189/196608   0x80000000a3405540

[11] CTD DLP FLOW :    65536/65536    0x80000000a5da5608

[12] CTD DLP DATA :     4096/4096     0x80000000a65e5688

[13] CTD DECODE FILTER :    65536/65536    0x80000000a69e9710

[14] Regex Results :     8000/8000     0x80000000a6d4a088

[15] TIMER Chunk :   131072/131072   0x80000000aeca5ae0

[16] FPTCP segs :    32768/32768    0x80000000b0d25b60

[17] Proxy session :     7936/7936     0x80000000b0dc5be0

[18] SSL Handshake State :     7936/7936     0x80000000b1096860

[19] SSL State :    15872/15872    0x80000000b19564e0

[20] SSL Handshake MAC State   : 17464/17464    0x80000000b25a0d60

[21] SSH Handshake State :       64/64 0x80000000b27d3ac0

[22] SSH State :      512/512 0x80000000b2841640

[23] TCP host connections :       15/16 0x80000000b297d020



show system statistics session

Device is up          : 7 days 0 hour 39 mins 39 sec

Packet rate           : 196/s

Throughput : 479 Kbps

Total active sessions : 818

Active TCP sessions   : 503

Active UDP sessions   : 311

Active ICMP sessions  : 4



show running resource-monitor

Resource monitoring sampling data (per second):

CPU load sampling by group:

flow_lookup :     5%

flow_fastpath :     5%

flow_slowpath :     5%

flow_forwarding :     5%

flow_mgmt :     1%

flow_ctrl :     1%

nac_result :     5%

flow_np : 5%

dfa_result :     5%

module_internal :     5%

aho_result :     5%

zip_result :     5%

pktlog_forwarding :     6%

lwm :     0%

flow_host :     1%


show session info

--------------------------------------------------------------------------------

Number of sessions supported: 262142

Number of active sessions: 845

Number of active TCP sessions: 505

Number of active UDP sessions: 330

Number of active ICMP sessions: 10

Number of active BCAST sessions: 0

Number of active MCAST sessions: 0

Number of active predict sessions: 0

Session table utilization: 0%

Number of sessions created since bootup:         4899529

Packet rate: 296/s

Throughput: 955 kbps

New connection establish rate: 0 cps

--------------------------------------------------------------------------------

Session timeout

TCP default timeout: 3600 secs

TCP session timeout before SYN-ACK received:      5 secs

TCP session timeout before 3-way handshaking:    10 secs

TCP session timeout after FIN/RST: 30 secs

UDP default timeout: 30 secs

ICMP default timeout: 6 secs

other IP default timeout: 30 secs

Captive Portal session timeout: 30 secs

Session timeout in discard state:

TCP: 90 secs, UDP: 60 secs, other IP protocols: 60 secs

--------------------------------------------------------------------------------

Session accelerated aging: True

Accelerated aging threshold: 80% of utilization

Scaling factor: 2 X

--------------------------------------------------------------------------------

Session setup

TCP - reject non-SYN first packet: True

Hardware session offloading: True

IPv6 firewalling: True

--------------------------------------------------------------------------------

Application trickling scan parameters:

Timeout to determine application trickling:    10 secs

Resource utilization threshold to start scan:  80%

Scan scaling factor over regular aging:        8

--------------------------------------------------------------------------------

Session behavior when resource limit is reached: drop

--------------------------------------------------------------------------------

Pcap token bucket rate : 10485760

--------------------------------------------------------------------------------



L7 Applicator

I would take a look at the rules and organization.  One of the things I've seen that can affect initial session setup time is the number of rules that are application only with the "any" port selection.

These rules should always be towards the bottom of the rule base.

The most specific rules with specified ports and applications should be at the top.

What happens with application and any rules is that PanOS essentially holds on that rule while the traffic is coming through and makes sure that the application is not a match before moving on to the next rule for checks.  If there are a number of these "any" port rules each one does the check and passes on and the time delay can add up.

Steve Puluka BSEET - IP Architect - DQE Communications (Metro Ethernet/ISP)
ACE PanOS 6; ACE PanOS 7; ASE 3.0; PSE 7.0 Foundations & Associate in Platform; Cyber Security; Data Center

show counter global filter delta yes  ( repeat the same command 5 times)


Global counters:

Elapsed time since last sampling: 156.760 seconds

name                                   value     rate severity  category  aspect    description

--------------------------------------------------------------------------------

pkt_recv                              569924     3635 info      packet    pktproc   Packets received

pkt_recv_zero                         569190     3630 info      packet    pktproc   Packets received from QoS 0

pkt_sent                              559573     3569 info      packet    pktproc   Packets transmitted

pkt_alloc                             588942     3756 info      packet    resource  Packets allocated

session_allocated                        846        5 info      session   resource  Sessions allocated

session_freed                            874        5 info      session   resource  Sessions freed

session_installed                        843        5 info      session   resource  Sessions installed

session_discard                            3        0 info      session   resource  Session set to discard by security policy check

flow_rcv_dot1q_tag_err                   247        1 drop      flow      parse     Packets dropped: 802.1q tag not configured

flow_no_interface                        247        1 drop      flow      parse     Packets dropped: invalid interface

flow_ipv6_disabled                       471        3 drop      flow      parse     Packets dropped: IPv6 disabled on interface

flow_policy_deny                         176        1 drop      flow      session   Session setup: denied by policy

flow_tcp_non_syn                          52        0 info      flow      session   Non-SYN TCP packets without session match

flow_tcp_non_syn_drop                     52        0 drop      flow      session   Packets dropped: non-SYN TCP without session match

flow_fwd_l3_bcast_drop                    18        0 drop      flow      forward   Packets dropped: unhandled IP broadcast

flow_fwd_l3_mcast_drop                   376        2 drop      flow      forward   Packets dropped: no route for IP multicast

flow_fwd_l3_noarp                         58        0 drop      flow      forward   Packets dropped: no ARP

flow_fwd_zonechange                       31        0 drop      flow      forward   Packets dropped: forwarded to different zone

flow_parse_unmatched_icmperr              77        0 info      flow      parse     Packets dropped: Unmatched ICMP error message

flow_dos_rule_allow_under_rate             6        0 info      flow      dos       Packets allowed: Rate within thresholds of DoS policy

flow_dos_rule_match                        6        0 info      flow      dos       Packets matched DoS policy

flow_dos_rule_nomatch                    840        5 info      flow      dos       Packets not matched DoS policy

flow_dos_ag_curr_sess_add_incr             6        0 info      flow      dos       Incremented aggregate current session count on session create

flow_dos_ag_curr_sess_del_decr             5        0 info      flow      dos       Decremented aggregate current session count on session delete

flow_dos_cl_curr_sess_add_incr             6        0 info      flow      dos       Incremented classified current session count on session create

flow_dos_cl_curr_sess_del_decr             5        0 info      flow      dos       Decremented classified current session count on session delete

flow_dos_ag_buckets_upd                  314        2 info      flow      dos       Updated aggregate buckets for aging

flow_action_close                          2        0 drop      flow      pktproc   TCP sessions closed via injecting RST

flow_arp_pkt_rcv                         178        1 info      flow      arp       ARP packets received

flow_arp_pkt_xmt                         122        0 info      flow      arp       ARP packets transmitted

flow_arp_pkt_replied                     114        0 info      flow      arp       ARP requests replied

flow_arp_rcv_gratuitous                   19        0 info      flow      arp       Gratuitous ARP packets received

flow_arp_resolve_xmt                      26        0 info      flow      arp       ARP resolution packets transmitted

flow_host_pkt_rcv                        734        4 info      flow      mgmt      Packets received from control plane

flow_host_pkt_xmt                       1437        9 info      flow      mgmt      Packets transmitted to control plane

flow_host_service_allow                   95        0 info      flow      mgmt      Device management session allowed

flow_health_monitor_rcv                  673        4 info      flow      mgmt      Health monitoring packet received

flow_health_monitor_xmt                  673        4 info      flow      mgmt      Health monitoring packet transmitted

appid_ident_by_icmp                      201        1 info      appid     pktproc   Application identified by icmp type

appid_ident_by_dport_first               156        0 info      appid     pktproc   Application identified by L4 dport first

appid_ident_by_dport                      12        0 info      appid     pktproc   Application identified by L4 dport

appid_proc                               443        2 info      appid     pktproc   The number of packets processed by Application identification

appid_use_dfa_1                          258        1 info      appid     pktproc   The number of packets using the second DFA table

appid_unknown_fini_empty                 109        0 info      appid     pktproc   The number of unknown applications because of no data

nat_static_xlat                            4        0 info      nat       resource  The total number of static NAT translate called

nat_static_release                         6        0 info      nat       resource  The total number of static NAT release called

nat_dynamic_port_xlat                     15        0 info      nat       resource  The total number of dynamic_ip_port NAT translate called

nat_dynamic_port_release                  15        0 info      nat       resource  The total number of dynamic_ip_port NAT release called

dfa_dte_request_total                 147094      938 info      dfa       offload   The total number of dfa match using dte

dfa_hte_in_cache_lookup               146276      933 info      dfa       offload   The total number of requests to an in cache HFA graph

tcp_case_2                               209        1 info      tcp       pktproc   tcp reassembly case 2

ctd_sml_exit_detector_i                   27        0 info      ctd       pktproc   The number of sessions with sml exit in detector i

appid_bypass_no_ctd                       12        0 info      appid     pktproc   appid bypass due to no ctd

ctd_err_bypass                            27        0 info      ctd       pktproc   ctd error bypass

ctd_run_pattern_match_failure             20        0 info      ctd       pktproc   Run pattern match failure

ctd_do_pattern_match                      20        0 info      ctd       pktproc   do pattern match

ctd_sml_vm_run_impl_opcodeexit            27        0 info      ctd       pktproc   SML VM opcode exit

[64;1H [K [7mlines 1-63  [27m [64;1H [64;1H [Kctd_sml_vm_run_impl_immed8000            662        4 info      ctd       pktproc   SML VM immed8000

ctd_sml_vm_check_domain                  159        1 info      ctd       pktproc   sml vm check domain

ctd_sml_opcode_set_file_type             688        4 info      ctd       pktproc   sml opcode set file type

ctd_filter_decode_failure_zip             34        0 error     ctd       pktproc   Number of decode filter failure for zip

ctd_bloom_filter_nohit                   162        1 info      ctd       pktproc   The number of no match for virus bloom filter

ctd_bloom_filter_hit                       9        0 info      ctd       pktproc   The number of match for virus bloom filter

ctd_bloom_filter_pattern_notfound          9        0 info      ctd       pktproc   The number of missing pattern match for virus bloom filter

aho_fpga                              297239     1896 info      aho       resource  The total requests to FPGA for AHO

aho_sw                                    97        0 info      aho       pktproc   The total usage of software for AHO

ctd_appid_reassign                     48318      308 info      ctd       pktproc   appid was changed

ctd_url_block                              1        0 info      ctd       pktproc   sessions blocked by url filtering

ctd_pkt_slowpath                      149601      954 info      ctd       pktproc   Packets processed by slowpath

ha_msg_sent                           290908     1855 info      ha        system    HA: messages sent

ha_msg_recv                              313        1 info      ha        system    HA: messages received

ha_session_setup_msg_sent                712        4 info      ha        pktproc   HA: session setup messages sent

ha_session_teardown_msg_sent             398        2 info      ha        pktproc   HA: session teardown messages sent

ha_session_update_msg_sent            289539     1847 info      ha        pktproc   HA: session update messages sent

ha_arp_update_msg_sent                   102        0 info      ha        pktproc   HA: ARP update messages sent

ha_ha2_monitor_msg_sent                  313        1 info      ha        pktproc   HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv                  313        1 info      ha        pktproc   HA: HA2 monitor message messages received

log_vulnerability_cnt                   2066       13 info      log       system    Number of vulnerability logs

log_fileext_cnt                           18        0 info      log       system    Number of file block logs

log_traffic_cnt                          980        6 info      log       system    Number of traffic logs

zip_process_total                        104        0 info      zip       pktproc   The total number of zip engine decompress process

zip_process_failure                       34        0 info      zip       pktproc   The number of failures for zip decompress process

pkt_nac_result                        297239     1896 info      packet    resource  Packets entered module nac stage result

pkt_flow_np                           271951     1734 info      packet    resource  Packets entered module flow stage np

pkt_flow_host                            734        4 info      packet    resource  Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 85

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 4.680 seconds

name                                   value     rate severity  category  aspect    description

--------------------------------------------------------------------------------

pkt_recv                               19543     4175 info      packet    pktproc   Packets received

pkt_recv_zero                          19522     4171 info      packet    pktproc   Packets received from QoS 0

pkt_sent                               17488     3736 info      packet    pktproc   Packets transmitted

pkt_alloc                              20124     4300 info      packet    resource  Packets allocated

session_allocated                         50       10 info      session   resource  Sessions allocated

session_freed                             18        3 info      session   resource  Sessions freed

session_installed                         50       10 info      session   resource  Sessions installed

flow_rcv_dot1q_tag_err                     6        1 drop      flow      parse     Packets dropped: 802.1q tag not configured

flow_no_interface                          6        1 drop      flow      parse     Packets dropped: invalid interface

flow_ipv6_disabled                        19        4 drop      flow      parse     Packets dropped: IPv6 disabled on interface

flow_policy_deny                           5        1 drop      flow      session   Session setup: denied by policy

flow_fwd_l3_bcast_drop                     2        0 drop      flow      forward   Packets dropped: unhandled IP broadcast

flow_fwd_l3_mcast_drop                    12        2 drop      flow      forward   Packets dropped: no route for IP multicast

flow_fwd_l3_noarp                          3        0 drop      flow      forward   Packets dropped: no ARP

flow_fwd_zonechange                        1        0 drop      flow      forward   Packets dropped: forwarded to different zone

flow_parse_unmatched_icmperr               6        1 info      flow      parse     Packets dropped: Unmatched ICMP error message

flow_dos_rule_nomatch                     50       10 info      flow      dos       Packets not matched DoS policy

flow_dos_ag_buckets_upd                    9        1 info      flow      dos       Updated aggregate buckets for aging

flow_arp_pkt_rcv                           5        1 info      flow      arp       ARP packets received

flow_arp_pkt_replied                       1        0 info      flow      arp       ARP requests replied

flow_host_pkt_rcv                         21        4 info      flow      mgmt      Packets received from control plane

flow_host_pkt_xmt                         39        8 info      flow      mgmt      Packets transmitted to control plane

flow_host_service_allow                    4        0 info      flow      mgmt      Device management session allowed

flow_health_monitor_rcv                   18        3 info      flow      mgmt      Health monitoring packet received

flow_health_monitor_xmt                   18        3 info      flow      mgmt      Health monitoring packet transmitted

appid_ident_by_icmp                       12        2 info      appid     pktproc   Application identified by icmp type

appid_ident_by_dport_first                 1        0 info      appid     pktproc   Application identified by L4 dport first

appid_proc                                30        6 info      appid     pktproc   The number of packets processed by Application identification

appid_use_dfa_1                           13        2 info      appid     pktproc   The number of packets using the second DFA table

appid_unknown_fini_empty                   7        1 info      appid     pktproc   The number of unknown applications because of no data

nat_static_xlat                            2        0 info      nat       resource  The total number of static NAT translate called

dfa_dte_request_total                   5534     1182 info      dfa       offload   The total number of dfa match using dte

dfa_hte_in_cache_lookup                 5465     1167 info      dfa       offload   The total number of requests to an in cache HFA graph

tcp_case_2                                 8        1 info      tcp       pktproc   tcp reassembly case 2

ctd_sml_exit_detector_i                    3        0 info      ctd       pktproc   The number of sessions with sml exit in detector i

appid_bypass_no_ctd                        3        0 info      appid     pktproc   appid bypass due to no ctd

ctd_err_bypass                             3        0 info      ctd       pktproc   ctd error bypass

ctd_run_pattern_match_failure              1        0 info      ctd       pktproc   Run pattern match failure

ctd_do_pattern_match                       1        0 info      ctd       pktproc   do pattern match

ctd_sml_vm_run_impl_opcodeexit             3        0 info      ctd       pktproc   SML VM opcode exit

ctd_sml_vm_run_impl_immed8000             20        4 info      ctd       pktproc   SML VM immed8000

ctd_sml_vm_check_domain                    1        0 info      ctd       pktproc   sml vm check domain

ctd_sml_opcode_set_file_type              21        4 info      ctd       pktproc   sml opcode set file type

ctd_bloom_filter_nohit                     1        0 info      ctd       pktproc   The number of no match for virus bloom filter

aho_fpga                               11017     2354 info      aho       resource  The total requests to FPGA for AHO

aho_sw                                    14        2 info      aho       pktproc   The total usage of software for AHO

ctd_appid_reassign                      1871      399 info      ctd       pktproc   appid was changed

ctd_pkt_slowpath                        5573     1190 info      ctd       pktproc   Packets processed by slowpath

ha_msg_sent                             9091     1942 info      ha        system    HA: messages sent

ha_msg_recv                               10        2 info      ha        system    HA: messages received

ha_session_setup_msg_sent                 40        8 info      ha        pktproc   HA: session setup messages sent

ha_session_teardown_msg_sent               8        1 info      ha        pktproc   HA: session teardown messages sent

ha_session_update_msg_sent              9037     1930 info      ha        pktproc   HA: session update messages sent

ha_arp_update_msg_sent                     1        0 info      ha        pktproc   HA: ARP update messages sent

ha_ha2_monitor_msg_sent                   10        2 info      ha        pktproc   HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv                   10        2 info      ha        pktproc   HA: HA2 monitor message messages received

log_vulnerability_cnt                     48       10 info      log       system    Number of vulnerability logs

[64;1H [K [7mlines 1-63  [27m [64;1H [64;1H [Klog_fileext_cnt                            4        0 info      log       system    Number of file block logs

log_traffic_cnt                           23        4 info      log       system    Number of traffic logs

pkt_nac_result                         11017     2354 info      packet    resource  Packets entered module nac stage result

pkt_flow_np                             8506     1817 info      packet    resource  Packets entered module flow stage np

pkt_flow_host                             21        4 info      packet    resource  Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 62

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 3.820 seconds

name                                   value     rate severity  category  aspect    description

--------------------------------------------------------------------------------

pkt_recv                               18988     4970 info      packet    pktproc   Packets received

pkt_recv_zero                          18967     4965 info      packet    pktproc   Packets received from QoS 0

pkt_sent                               16860     4413 info      packet    pktproc   Packets transmitted

pkt_alloc                              19541     5115 info      packet    resource  Packets allocated

session_allocated                         50       13 info      session   resource  Sessions allocated

session_freed                             18        4 info      session   resource  Sessions freed

session_installed                         50       13 info      session   resource  Sessions installed

flow_rcv_dot1q_tag_err                     7        1 drop      flow      parse     Packets dropped: 802.1q tag not configured

flow_no_interface                          7        1 drop      flow      parse     Packets dropped: invalid interface

flow_ipv6_disabled                        12        3 drop      flow      parse     Packets dropped: IPv6 disabled on interface

flow_policy_deny                           6        1 drop      flow      session   Session setup: denied by policy

flow_tcp_non_syn                           1        0 info      flow      session   Non-SYN TCP packets without session match

flow_tcp_non_syn_drop                      1        0 drop      flow      session   Packets dropped: non-SYN TCP without session match

flow_fwd_l3_mcast_drop                     9        2 drop      flow      forward   Packets dropped: no route for IP multicast

flow_fwd_zonechange                        1        0 drop      flow      forward   Packets dropped: forwarded to different zone

flow_dos_rule_allow_under_rate             1        0 info      flow      dos       Packets allowed: Rate within thresholds of DoS policy

flow_dos_rule_match                        1        0 info      flow      dos       Packets matched DoS policy

flow_dos_rule_nomatch                     49       12 info      flow      dos       Packets not matched DoS policy

flow_dos_ag_curr_sess_add_incr             1        0 info      flow      dos       Incremented aggregate current session count on session create

flow_dos_cl_curr_sess_add_incr             1        0 info      flow      dos       Incremented classified current session count on session create

flow_dos_ag_buckets_upd                    8        2 info      flow      dos       Updated aggregate buckets for aging

flow_arp_pkt_rcv                           3        0 info      flow      arp       ARP packets received

flow_arp_pkt_replied                       3        0 info      flow      arp       ARP requests replied

flow_host_pkt_rcv                         21        5 info      flow      mgmt      Packets received from control plane

flow_host_pkt_xmt                         45       11 info      flow      mgmt      Packets transmitted to control plane

flow_host_service_allow                    3        0 info      flow      mgmt      Device management session allowed

flow_health_monitor_rcv                   19        4 info      flow      mgmt      Health monitoring packet received

flow_health_monitor_xmt                   19        4 info      flow      mgmt      Health monitoring packet transmitted

appid_ident_by_icmp                        8        2 info      appid     pktproc   Application identified by icmp type

appid_ident_by_dport_first                16        4 info      appid     pktproc   Application identified by L4 dport first

appid_proc                                27        7 info      appid     pktproc   The number of packets processed by Application identification

appid_use_dfa_1                           12        3 info      appid     pktproc   The number of packets using the second DFA table

appid_unknown_fini_empty                   3        0 info      appid     pktproc   The number of unknown applications because of no data

dfa_dte_request_total                   5339     1397 info      dfa       offload   The total number of dfa match using dte

dfa_hte_in_cache_lookup                 5309     1389 info      dfa       offload   The total number of requests to an in cache HFA graph

tcp_case_2                                 7        1 info      tcp       pktproc   tcp reassembly case 2

ctd_sml_vm_run_impl_immed8000             19        4 info      ctd       pktproc   SML VM immed8000

ctd_sml_vm_check_domain                   16        4 info      ctd       pktproc   sml vm check domain

ctd_sml_opcode_set_file_type              19        4 info      ctd       pktproc   sml opcode set file type

ctd_filter_decode_failure_zip              7        1 error     ctd       pktproc   Number of decode filter failure for zip

ctd_bloom_filter_nohit                    16        4 info      ctd       pktproc   The number of no match for virus bloom filter

aho_fpga                               10766     2818 info      aho       resource  The total requests to FPGA for AHO

aho_sw                                     2        0 info      aho       pktproc   The total usage of software for AHO

ctd_appid_reassign                      1814      474 info      ctd       pktproc   appid was changed

ctd_pkt_slowpath                        5435     1422 info      ctd       pktproc   Packets processed by slowpath

ha_msg_sent                             8751     2290 info      ha        system    HA: messages sent

ha_msg_recv                                8        2 info      ha        system    HA: messages received

ha_session_setup_msg_sent                 45       11 info      ha        pktproc   HA: session setup messages sent

ha_session_teardown_msg_sent               8        2 info      ha        pktproc   HA: session teardown messages sent

ha_session_update_msg_sent              8691     2275 info      ha        pktproc   HA: session update messages sent

ha_arp_update_msg_sent                     3        0 info      ha        pktproc   HA: ARP update messages sent

ha_ha2_monitor_msg_sent                    8        2 info      ha        pktproc   HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv                    8        2 info      ha        pktproc   HA: HA2 monitor message messages received

log_url_req_cnt                            2        0 info      log       system    Number of url request logs

log_vulnerability_cnt                     71       18 info      log       system    Number of vulnerability logs

log_traffic_cnt                           23        6 info      log       system    Number of traffic logs

url_db_request                             2        0 info      url       pktproc   Number of URL database request

[64;1H [K [7mlines 1-63  [27m [64;1H [64;1H [Kurl_db_reply                               2        0 info      url       pktproc   Number of URL reply

zip_process_total                         23        6 info      zip       pktproc   The total number of zip engine decompress process

zip_process_failure                        7        1 info      zip       pktproc   The number of failures for zip decompress process

pkt_nac_result                         10766     2818 info      packet    resource  Packets entered module nac stage result

pkt_flow_np                             8200     2146 info      packet    resource  Packets entered module flow stage np

pkt_flow_host                             21        5 info      packet    resource  Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 63

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 4.722 seconds

name                                   value     rate severity  category  aspect    description

--------------------------------------------------------------------------------

pkt_recv                               18897     4001 info      packet    pktproc   Packets received

pkt_recv_zero                          18875     3997 info      packet    pktproc   Packets received from QoS 0

pkt_sent                               16528     3500 info      packet    pktproc   Packets transmitted

pkt_alloc                              19459     4120 info      packet    resource  Packets allocated

session_allocated                         33        6 info      session   resource  Sessions allocated

session_freed                             39        8 info      session   resource  Sessions freed

session_installed                         33        6 info      session   resource  Sessions installed

session_discard                            1        0 info      session   resource  Session set to discard by security policy check

flow_rcv_dot1q_tag_err                     9        1 drop      flow      parse     Packets dropped: 802.1q tag not configured

flow_no_interface                          9        1 drop      flow      parse     Packets dropped: invalid interface

flow_ipv6_disabled                        14        2 drop      flow      parse     Packets dropped: IPv6 disabled on interface

flow_policy_deny                           2        0 drop      flow      session   Session setup: denied by policy

flow_fwd_l3_mcast_drop                    13        2 drop      flow      forward   Packets dropped: no route for IP multicast

flow_fwd_l3_noarp                          3        0 drop      flow      forward   Packets dropped: no ARP

flow_fwd_zonechange                        1        0 drop      flow      forward   Packets dropped: forwarded to different zone

flow_parse_unmatched_icmperr               6        1 info      flow      parse     Packets dropped: Unmatched ICMP error message

flow_dos_rule_nomatch                     33        6 info      flow      dos       Packets not matched DoS policy

flow_dos_ag_buckets_upd                    9        1 info      flow      dos       Updated aggregate buckets for aging

flow_arp_pkt_rcv                           4        0 info      flow      arp       ARP packets received

flow_arp_pkt_xmt                           1        0 info      flow      arp       ARP packets transmitted

flow_arp_pkt_replied                       2        0 info      flow      arp       ARP requests replied

flow_arp_resolve_xmt                       1        0 info      flow      arp       ARP resolution packets transmitted

flow_host_pkt_rcv                         22        4 info      flow      mgmt      Packets received from control plane

flow_host_pkt_xmt                         43        9 info      flow      mgmt      Packets transmitted to control plane

flow_host_service_allow                    1        0 info      flow      mgmt      Device management session allowed

flow_health_monitor_rcv                   21        4 info      flow      mgmt      Health monitoring packet received

flow_health_monitor_xmt                   21        4 info      flow      mgmt      Health monitoring packet transmitted

appid_ident_by_icmp                        9        1 info      appid     pktproc   Application identified by icmp type

appid_ident_by_dport_first                 5        1 info      appid     pktproc   Application identified by L4 dport first

appid_proc                                15        3 info      appid     pktproc   The number of packets processed by Application identification

appid_use_dfa_1                           13        2 info      appid     pktproc   The number of packets using the second DFA table

appid_unknown_fini_empty                  10        2 info      appid     pktproc   The number of unknown applications because of no data

nat_dynamic_port_xlat                      2        0 info      nat       resource  The total number of dynamic_ip_port NAT translate called

dfa_dte_request_total                   5384     1140 info      dfa       offload   The total number of dfa match using dte

dfa_hte_in_cache_lookup                 5361     1135 info      dfa       offload   The total number of requests to an in cache HFA graph

tcp_case_2                                 3        0 info      tcp       pktproc   tcp reassembly case 2

ctd_sml_exit_detector_i                    1        0 info      ctd       pktproc   The number of sessions with sml exit in detector i

appid_bypass_no_ctd                        1        0 info      appid     pktproc   appid bypass due to no ctd

ctd_err_bypass                             1        0 info      ctd       pktproc   ctd error bypass

ctd_run_pattern_match_failure              1        0 info      ctd       pktproc   Run pattern match failure

ctd_do_pattern_match                       1        0 info      ctd       pktproc   do pattern match

ctd_sml_vm_run_impl_opcodeexit             1        0 info      ctd       pktproc   SML VM opcode exit

ctd_sml_vm_run_impl_immed8000             20        4 info      ctd       pktproc   SML VM immed8000

ctd_sml_vm_check_domain                    3        0 info      ctd       pktproc   sml vm check domain

ctd_sml_opcode_set_file_type              21        4 info      ctd       pktproc   sml opcode set file type

ctd_bloom_filter_nohit                     3        0 info      ctd       pktproc   The number of no match for virus bloom filter

aho_fpga                               10839     2295 info      aho       resource  The total requests to FPGA for AHO

aho_sw                                     4        0 info      aho       pktproc   The total usage of software for AHO

ctd_appid_reassign                      1798      380 info      ctd       pktproc   appid was changed

ctd_url_block                              1        0 info      ctd       pktproc   sessions blocked by url filtering

ctd_pkt_slowpath                        5447     1153 info      ctd       pktproc   Packets processed by slowpath

ha_msg_sent                             8598     1820 info      ha        system    HA: messages sent

ha_msg_recv                                9        1 info      ha        system    HA: messages received

ha_session_setup_msg_sent                 24        5 info      ha        pktproc   HA: session setup messages sent

ha_session_teardown_msg_sent               7        1 info      ha        pktproc   HA: session teardown messages sent

ha_session_update_msg_sent              8560     1812 info      ha        pktproc   HA: session update messages sent

ha_arp_update_msg_sent                     3        0 info      ha        pktproc   HA: ARP update messages sent

[64;1H [K [7mlines 1-63  [27m [64;1H [64;1H [Kha_ha2_monitor_msg_sent                    9        1 info      ha        pktproc   HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv                    9        1 info      ha        pktproc   HA: HA2 monitor message messages received

log_vulnerability_cnt                     69       14 info      log       system    Number of vulnerability logs

log_traffic_cnt                           33        6 info      log       system    Number of traffic logs

pkt_nac_result                         10839     2295 info      packet    resource  Packets entered module nac stage result

pkt_flow_np                             8036     1701 info      packet    resource  Packets entered module flow stage np

pkt_flow_host                             22        4 info      packet    resource  Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 64

--------------------------------------------------------------------------------

Global counters:

Elapsed time since last sampling: 3.18  seconds

name                                   value     rate severity  category  aspect    description

--------------------------------------------------------------------------------

pkt_recv                                1006      333 info      packet    pktproc   Packets received

pkt_recv_zero                            994      329 info      packet    pktproc   Packets received from QoS 0

pkt_sent                                1832      607 info      packet    pktproc   Packets transmitted

pkt_alloc                               1075      356 info      packet    resource  Packets allocated

session_allocated                         13        4 info      session   resource  Sessions allocated

session_freed                             15        4 info      session   resource  Sessions freed

session_installed                         13        4 info      session   resource  Sessions installed

flow_rcv_dot1q_tag_err                     4        1 drop      flow      parse     Packets dropped: 802.1q tag not configured

flow_no_interface                          4        1 drop      flow      parse     Packets dropped: invalid interface

flow_ipv6_disabled                         6        1 drop      flow      parse     Packets dropped: IPv6 disabled on interface

flow_policy_deny                           3        0 drop      flow      session   Session setup: denied by policy

flow_fwd_l3_mcast_drop                     9        2 drop      flow      forward   Packets dropped: no route for IP multicast

flow_fwd_l3_noarp                          1        0 drop      flow      forward   Packets dropped: no ARP

flow_parse_unmatched_icmperr               1        0 info      flow      parse     Packets dropped: Unmatched ICMP error message

flow_dos_rule_allow_under_rate             1        0 info      flow      dos       Packets allowed: Rate within thresholds of DoS policy

flow_dos_rule_match                        1        0 info      flow      dos       Packets matched DoS policy

flow_dos_rule_nomatch                     12        3 info      flow      dos       Packets not matched DoS policy

flow_dos_ag_curr_sess_add_incr             1        0 info      flow      dos       Incremented aggregate current session count on session create

flow_dos_cl_curr_sess_add_incr             1        0 info      flow      dos       Incremented classified current session count on session create

flow_dos_ag_buckets_upd                    6        1 info      flow      dos       Updated aggregate buckets for aging

flow_arp_pkt_rcv                           3        0 info      flow      arp       ARP packets received

flow_arp_pkt_replied                       1        0 info      flow      arp       ARP requests replied

flow_arp_rcv_gratuitous                    1        0 info      flow      arp       Gratuitous ARP packets received

flow_host_pkt_rcv                         12        3 info      flow      mgmt      Packets received from control plane

flow_host_pkt_xmt                         19        6 info      flow      mgmt      Packets transmitted to control plane

flow_host_service_allow                    2        0 info      flow      mgmt      Device management session allowed

flow_health_monitor_rcv                   11        3 info      flow      mgmt      Health monitoring packet received

flow_health_monitor_xmt                   11        3 info      flow      mgmt      Health monitoring packet transmitted

appid_ident_by_icmp                        4        1 info      appid     pktproc   Application identified by icmp type

appid_proc                                 9        2 info      appid     pktproc   The number of packets processed by Application identification

appid_use_dfa_1                            7        2 info      appid     pktproc   The number of packets using the second DFA table

dfa_dte_request_total                    106       35 info      dfa       offload   The total number of dfa match using dte

dfa_hte_in_cache_lookup                   99       32 info      dfa       offload   The total number of requests to an in cache HFA graph

ctd_sml_vm_run_impl_immed8000             12        3 info      ctd       pktproc   SML VM immed8000

ctd_sml_opcode_set_file_type              15        4 info      ctd       pktproc   sml opcode set file type

aho_fpga                                  85       28 info      aho       resource  The total requests to FPGA for AHO

ctd_pkt_slowpath                         102       33 info      ctd       pktproc   Packets processed by slowpath

ha_msg_sent                              984      326 info      ha        system    HA: messages sent

ha_msg_recv                                6        1 info      ha        system    HA: messages received

ha_session_setup_msg_sent                 11        3 info      ha        pktproc   HA: session setup messages sent

ha_session_teardown_msg_sent               7        2 info      ha        pktproc   HA: session teardown messages sent

ha_session_update_msg_sent               962      318 info      ha        pktproc   HA: session update messages sent

ha_arp_update_msg_sent                     1        0 info      ha        pktproc   HA: ARP update messages sent

ha_ha2_monitor_msg_sent                    6        1 info      ha        pktproc   HA: HA2 monitor message messages sent

ha_ha2_monitor_msg_recv                    6        1 info      ha        pktproc   HA: HA2 monitor message messages received

log_traffic_cnt                           21        6 info      log       system    Number of traffic logs

pkt_nac_result                            85       28 info      packet    resource  Packets entered module nac stage result

pkt_flow_np                              909      301 info      packet    resource  Packets entered module flow stage np

pkt_flow_host                             12        3 info      packet    resource  Packets entered module flow stage host

--------------------------------------------------------------------------------

Total counters shown: 49

--------------------------------------------------------------------------------

Thanks for your input!

In this firewall, we do not have any rules which have an application specified along with service ANY. We do however have several rules which list an application with service "application-default". Is it faster processing wise to explicitly list the port as oppose to app default?

Hello Mackwage,

Output looks good, no issue. Would it be possible to provide packet captures.

Regards,

Hardik Shah

Will get those to you shortly. One question on the buffer output...

Hardware Pools

[ 0] Packet Buffers :    11341/11468    0x8000000031000000

[ 1] Work Queue Entries        : 229309/229376  0x8000000037ffe000

[ 2] Output Buffers :    1006/1024    0x8000000039bfe000

[ 3] DFA Result :    3999/4000    0x8000000039cfe000

[ 4] Timer Buffers :    4092/4096    0x800000003a0e6000

[ 5] PAN_FPA_LWM_POOL :    1024/1024    0x8000000000e9f200

[ 6] PAN_FPA_ZIP_POOL :    1023/1024    0x800000003a4e6000

[ 7] PAN_FPA_BLAST_POOL :    1024/1024    0x800000000ff00000

Software Pools

[ 0] software packet buffer 0  : 32767/32768    0x800000003a6e6680

[ 1] software packet buffer 1  : 32768/32768    0x800000003b706700

[ 2] software packet buffer 2  : 81920/81920    0x800000003d726780

[ 3] software packet buffer 3  : 20480/20480    0x8000000047776800


The buffers say for example 32767/32768. Does this imply the buffer is almost full? Or is it specifying it is empty?

Buffers are empty, buffers are not utilized, buffers are under utilized.

There are some drop counters..

flow_policy_deny                           6        1 drop      flow      session   Session setup: denied by policy

flow_tcp_non_syn                           1        0 info      flow      session   Non-SYN TCP packets without session match

flow_tcp_non_syn_drop                      1        0 drop      flow      session   Packets dropped: non-SYN TCP without session match

flow_fwd_l3_mcast_drop                     9        2 drop      flow      forward   Packets dropped: no route for IP multicast

flow_fwd_zonechange                        1        0 drop      flow      forward   Packets dropped: forwarded to different zone

Could you configure a packet-filter based on the source and destination used for testing and collect new counters?

> debug dataplane packet-diag set filter match source x.x.x.x destination y.y.y.y

> debug dataplane packet-diag set filter on

Then collect counters with packet-filter

> show counter global name filter packet-filter yes delta yes

*Note* the filter will only match new sessions, not sessions already present in session table

I have the pcaps but I am not seeing a forum option to upload attachments. :smileylaugh:

mackwage wrote:

Thanks for your input!

In this firewall, we do not have any rules which have an application specified along with service ANY. We do however have several rules which list an application with service "application-default". Is it faster processing wise to explicitly list the port as oppose to app default?

I don't have any direct experience with that situation.  I would assume that it would only affect your target traffic if one of the default application ports overlapped with your delayed traffic.

Steve Puluka BSEET - IP Architect - DQE Communications (Metro Ethernet/ISP)
ACE PanOS 6; ACE PanOS 7; ASE 3.0; PSE 7.0 Foundations & Associate in Platform; Cyber Security; Data Center
  • 6332 Views
  • 17 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!