Debugging packet flow.

[PKCCommsTeam]

Hi there,

We have just moved from a Juniper SSG-550 with around 700 policies to a PaloAlto 3050.

Naturally this has thrown up a few issues!

Can anyone explain how to do the equivalent of a Juniper “debug flow basic” on the PaloAlto?

On the Juniper, this would all you to follow the journey of a packet from ingress to egress, through the entire decision making progress of the firewall as it processed the packet making troubleshooting very simple.

I’m finding that packet captures, “test” commands and “debug dataplane packet-diag set log feature flow basic” on the PaloAlto a little erratic.

To do the dataplane flow debug I’m following instructions here:

https://live.paloaltonetworks.com/t5/Management-Articles/Packet-Capture-Debug-Flow-basic-and-Counter...

However, i'm finding that filters do not appear to be getting applied correctly.

More often than not the logs appear to show no traffic filter is in effect and everything is being logged, yet "debug dataplane packet-diag show setting" shows that the packet filter is enabled, and correctly configured.

Can anyone help?

We are running v7.0.3.

Many thanks,

Mark.

[glastra1]

what do you mean by "erratic"? 

If you had a filter before and the sessions tagged by the previous filter are still generating traffic you'll se those packets in the debug, 

>debug dataplane packet-diag clear filter-marked-session all

As you said before the equivalent is

> debug dataplane packet-diag set log feature flow basic 
> debug dataplane packet-diag set log on 

After that you just aggregate logs and check them as explained in the KB.

Regards,

Gerardo. 

[PKCCommsTeam]

Hi there, thanks for the reply.

Debug is set up as follows, (session offload is turned off):

PA-3050-A(active)> debug dataplane packet-diag show setting

--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   yes
  Match pre-parsed packet:   yes
  Index 1: 10.20.11.173[0]->0.0.0.0[0], proto 0
           ingress-interface any, egress-interface any, exclude non-IP
  Index 2: 0.0.0.0[0]->10.20.11.173[0], proto 0
           ingress-interface any, egress-interface any, exclude non-IP
--------------------------------------------------------------------------------
Logging
  Enabled:                   no
  Log-throttle:              no
  Sync-log-by-ticks:         yes
  Features:
    flow    : basic
  Counters:
--------------------------------------------------------------------------------
Packet capture
  Enabled:                   no
  Snaplen:                   0
--------------------------------------------------------------------------------
PA-3050-A(active)>           

I clear the logs, then verify nothing is there:

PA-3050-A(active)> debug dataplane packet-diag clear log log
dataplane debug logs cleared

PA-3050-A(active)> less dp-log pan_packet_diag.log
0%
PA-3050-A(active)>     

I then enable the flow debug, make some traffic, disable debug, wait a while then aggregate logs:

PA-3050-A(active)> debug dataplane packet-diag clear log log
dataplane debug logs cleared

PA-3050-A(active)> less dp-log pan_packet_diag.log

PA-3050-A(active)> debug dataplane packet-diag set log on
Packet log is enabled

PA-3050-A(active)> debug dataplane packet-diag set log off
Packet log is disabled

PA-3050-A(active)> debug dataplane packet-diag aggregate-logs
packet-diag.log is aggregated

PA-3050-A(active)> 

Then have a look at the logs, which are many hundreds of lines long matching traffic completley unrelated to the filter:

2015-11-23 13:17:55.332 +0000 debug: pan_cfg_nat_ruledata_lookup(pan_cfg_nat_policy.c:252): Rule: index=106 name=Default Outbound, cfg_pool_idx=53 cfg_fallback_pool_idx=0
2015-11-23 13:17:55.332 +0000 debug: pan_flow_nat_setup_session(src/pan_flow_nat.c:2128): NAT Rule: name=Default Outbound, cfg_pool_idx=53; Session: index=491969, nat_pool_idx=53

2015-11-23 13:17:55.332 +0000 Error:  pan_cfg_app_policy_lookup_i(pan_cfg_policy.c:323): no rule is matched

2015-11-23 13:17:55.333 +0000 Error:  pan_cfg_app_policy_lookup_i(pan_cfg_policy.c:323): no rule is matched
2015-11-23 13:17:55.333 +0000 Error:  pan_appid_finish(pan_appid_proc.c:2454): pan_appid_policy_lookup() failed                   

== 2015-11-23 13:17:58.024 +0000 ==
Packet received at fastpath stage
Packet info: len 64 port 48 interface 256 vsys 1
  wqe index 223936 packet 0x0x800000003228b0e2
Packet decoded dump:
L2:     20:b3:99:57:98:f5->00:1b:17:00:01:30, VLAN 110 (0x8100 0x006e), type 0x0800
IP:     192.168.69.149->192.168.215.14, protocol 17
        version 4, ihl 5, tos 0x00, len 45,
        id 3483, frag_off 0x0000, ttl 126, checksum 37168
UDP:    sport 49336, dport 47808, len 25, checksum 15361
Flow fastpath, session 161631
Forwarding lookup, ingress interface 256
L3 mode, virtual-router 8
Route lookup in virtual-router 8, IP 192.168.215.14
Route found, interface ae2.1, zone 18, nexthop 192.168.249.142
Resolve ARP for IP 192.168.249.142 on interface ae2.1
ARP entry found on interface 260
Transmit packet on port 49         

However i do actually find what i'm looking for:

== 2015-11-23 13:18:10.113 +0000 ==
Packet received at ingress stage
Packet info: len 66 port 49 interface 292 vsys 1
  wqe index 222463 packet 0x0x80000000344ce0e2
Packet decoded dump:
L2:     20:b3:99:57:98:f3->00:1b:17:00:01:31, VLAN 135 (0x8100 0x0087), type 0x0800
IP:     10.20.11.173->*.*.*.*, protocol 6
        version 4, ihl 5, tos 0x20, len 48,
        id 7477, frag_off 0x4000, ttl 127, checksum 51783
TCP:    sport 51822, dport 443, seq 686207197, ack 0,
        reserved 0, offset 7, window 8192, checksum 43334,
        flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 74 01 01 04 02                             ...t.... 
Flow lookup, key word0 0xca6e01bb000a0600 word1 0
Session setup: vsys 1
No active flow found, enqueue to create session


== 2015-11-23 13:18:10.113 +0000 ==
Packet received at slowpath stage
Packet info: len 66 port 49 interface 292 vsys 1
  wqe index 222463 packet 0x0x80000000344ce0e2
Packet decoded dump:
L2:     20:b3:99:57:98:f3->00:1b:17:00:01:31, VLAN 135 (0x8100 0x0087), type 0x0800
IP:     10.20.11.173->*.*.*.*, protocol 6
        version 4, ihl 5, tos 0x20, len 48,
        id 7477, frag_off 0x4000, ttl 127, checksum 51783
TCP:    sport 51822, dport 443, seq 686207197, ack 0,
        reserved 0, offset 7, window 8192, checksum 43334,
        flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 74 01 01 04 02                             ...t.... 
Session setup: vsys 1
PBF lookup (vsys 1) with application none
Packet matches PBF policy (vsys 1) rule idx 3 (id 15) for c2s.
67%PBF policy rule 3 is NO-PBF.
Packet dropped, no route during session setup
Packet dropped, forwarding info unavailable for policy lookup
Packet dropped, Session setup failed

So this is great, but why do i get hundreds of lines in the log relating to packets that don't match the filter?

I can't find a way to SCP this log file off the box either - you only seem to be able to output it to the screen.

Many thanks,

Mark.

[Brandon_Wertz]

While I can't provide any specific answers to your questions.  Given you're relatively new to the product I might be able to point you to a different T/S method.  What's the issue(s) you're trying to tackle?

[reaper]

Hi

you'll want to start with disabling pre-parse match, this will basically bypass your filter as it tells your packet-diag to listen for everything before it hits any logic, which includes your filters. this is why you're seeing so much unrelated logs:

> debug dataplane packet-diag set filter pre-parse-match no

next, you will want to clear any sessions that already exist as an existing session will not show up in flow basic or packetcaptures 

> clear session all filter source 10.20.11.173
sessions cleared
> clear session all filter destination 10.20.11.173
sessions cleared

depending on the kind of traffic you're trying to capture, it may get offloaded at which point flow basic will not be able to log additional packets, so you can disable offloading if required:

> set session offload no

(make sure to re-enable offloading after you're done)

hope this helps!

Tom

[ghostrider]

Hello Tom

In PA doucmentation, no where it is mentioned that for filter to work with debug, you have to use below command:

> debug dataplane packet-diag set filter pre-parse-match no

For exmaple in below doc, no where above command is mentioned

https://live.paloaltonetworks.com/t5/Management-Articles/Packet-Capture-Debug-Flow-basic-and-Counter...

[reaper]

Hi @ghostrider

That is because it is set to 'no' by default (so normally you do not need this command)

I mentioned this because OP had set it to yes (manually) which will cause the filter to not be applied

hope this helps

[Douglas_Zan]

Does anybody know if when I disable the session offload, it disables the offloading for new sessions or also for active sessions?  

[reaper]

It only disables offloading for new sessions, so you may need to clear some sessions to get the desired result

[OtakarKlier]

Hello,

Also please upgrade the OS of the device, its old and has known vulnerabilities!

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClTJCA0

Regards,