topic Re: Application Incomplete - Leading causes? in General Topics

Application Incomplete - Leading causes?

choff123 — Fri, 27 Sep 2013 20:38:58 GMT

So im doing work on our DR site. Two diff setup scenarios are failing; NAT over a VPN and routing from one PA to another and out a VPN. I can see the rule letting the packets out so a session should start for the return trip but .. nothing.

Both scenarios show Application incomplete; what are the leading causes of this? Incomplete handshake? So maybe the other side never returns an ack on the tcp handshake? Return routing does not work?

Any inout would be appreciated as im running out of options.

thanks

Re: Application Incomplete - Leading causes?

Retired Member — Fri, 27 Sep 2013 20:44:39 GMT

You should take packet capture by filter both source and destinaion ip address.This will make sure you to see if there is a drop or no return.

Re: Application Incomplete - Leading causes?

choff123 — Fri, 27 Sep 2013 20:49:33 GMT

ok, thanks, ill try that

Re: Application Incomplete - Leading causes?

zarina — Fri, 27 Sep 2013 23:35:38 GMT

You can find the definition for incomplete here: https://live.paloaltonetworks.com/docs/DOC-1549

As for the traffic, you can also look at the sessions to see if there are any packets from server to client. You can view this data from GUI through Monitor -> Traffic logs -> Click on the magnifying glass and look for packets sent and packets received. From cli you can do: show session all filter <interesting traffic> and show session id <id>. You can also look at the global counters to see if there are any drops.

You can alos refer to these documents for troubleshooting using counters & captures:

https://live.paloaltonetworks.com/docs/DOC-2542

https://live.paloaltonetworks.com/docs/DOC-2310

Re: Application Incomplete - Leading causes?

gswcowboy — Sat, 28 Sep 2013 04:40:27 GMT

Incomplete denotes a lack of a tcp handshake. Syn packets allowed but never getting a syn-ack from the destination device. Perform client pcaps and span port on switch to help determine where those syn-acks are going if in fact the server is sending them.

Re: Application Incomplete - Leading causes?

gafrol — Mon, 30 Sep 2013 09:29:34 GMT

You mention DR site. Are you facing asynchronous routing ?

Re: Application Incomplete - Leading causes?

nayubi — Mon, 30 Sep 2013 17:58:56 GMT

This can be due to not receiving enough packets for the decoder to identify the application. Depending on what application it is, it may need more packets to identify.

Re: Application Incomplete - Leading causes?

choff123 — Mon, 30 Sep 2013 18:41:57 GMT

so i did a pcap.

seems the 'drop' log has a bunch of packets in it (outbound to server). So i guess they are being dropped on the way out on the initial connection attempt?

the transmit log is empty so i guess my packets are not even leaving the Firewall

however, if i test the fw rule from the command line everything seems fine (action allow).

test security-policy-match from ZONE source <My IP> to ZONE destination <Remote IP> protocol 6 destination-port 23

also, the rule as it stands right now is very loose.

@zarnia: Ill try looking at session data next,

@gafrol: I dont think there is a routing issue. The client LAN hooks up to the PA, then out through the IPSec VPN on the PA, so there not much room for different routes if any at all.

LAN -> PA/NAT -> IPSec VPN -> Remote Side

Re: Application Incomplete - Leading causes?

gswcowboy — Mon, 30 Sep 2013 19:35:33 GMT

We're dropping packets? If you're good with flow basic, you can get granular information on why we're dropping? If not, please call into PAN Support as any debugging not done properly can impact services on or through the PAN device.

Re: Application Incomplete - Leading causes?

mbutt — Mon, 30 Sep 2013 20:09:17 GMT

Here are some good tips on debugging packet drops

1. Need to setup the filters for the traffic we are interested in. To do this, execute the following steps:

Navigate to Monitor--Packet Capture

Click 'Manage Filters'

Set Filter ID 1 to be the source IP and destination IP of traffic you feel is affected ( leave all other fields blank )

Set Filter ID 2 to be the exact inverse of what you did in step 3 (destination IP in source field, Source IP in destination field)

2. Setup up the captures

Create and name the file stage for a packet capture on all the stages (receive, transmit, firewall and drop)

3. Enable filters and captures

debug dataplane packet-diag set filter on

debug dataplane packet-diag set capture on

4. open 2 CLI windows

on 1 run the following command to look at the counter ( make sure it run this command once before running the traffic)

show counter global filter packet-filter yes delta yes

on the 2nd window run the following command to look at he sessions

show session all filter source <ip address> destination <ip address>

After your test has been done stop all the captures and filters and see if global counter show you anything why it is dropping the traffic or if you have getting pcap with drop stage.

This will help you narrow down the issue.

Let us know if this helps you resolve the issue.

Thanks

Numan

Re: Application Incomplete - Leading causes?

choff123 — Mon, 30 Sep 2013 21:03:54 GMT

thanks fo the advice.

I cant see anything on the output thats helps so far

..and there was nothing in the session window, no session at all, so i guess the handshake never completes, i take it we can confirm the handshake never completes.

Re: Application Incomplete - Leading causes?

gswcowboy — Mon, 30 Sep 2013 21:58:56 GMT

sessions won't be built if we're dropping packets. global counters should be run consistently/repeatedly w/ hope of determining reason for dropped packets. The last screen shot unfortunately doesn't give you nor us insight.

Re: Application Incomplete - Leading causes?

choff123 — Mon, 30 Sep 2013 22:10:16 GMT

ok, well, ill try harder to catch something with the counters. If I cant catch anything, ill call in, thanks for the help.

Re: Application Incomplete - Leading causes?

gswcowboy — Mon, 30 Sep 2013 22:29:13 GMT

Hopefully those delta counters can give us a clue. Focus on dropped packets via the following:

admin@Phoenix-VM-Lab148> show counter global filter delta yes packet-filter yes severity drop

Global counters:

Elapsed time since last sampling: 3.307 seconds

name value rate severity category aspect description

--------------------------------------------------------------------------------

flow_rcv_dot1q_tag_err 627 189 drop flow parse Packets dropped: 802.1q tag not configured

flow_no_interface 627 189 drop flow parse Packets dropped: invalid interface

flow_host_service_deny 1 0 drop flow mgmt Device management session denied

--------------------------------------------------------------------------------

Total counters shown: 3

-------------------------------------------------------------------

Thanks!