Application Incomplete - Leading causes?

Reply
Highlighted
L3 Networker

Application Incomplete - Leading causes?

So im doing work on our DR site. Two diff setup scenarios are failing; NAT over a VPN and routing from one PA to another and out a VPN. I can see the rule letting the packets out so a session should start for the return trip but .. nothing.

Both scenarios show Application incomplete; what are the leading causes of this? Incomplete handshake? So maybe the other side never returns an ack on the tcp handshake? Return routing does not work?

Any inout would be appreciated as im running out of options.

thanks

Tags (1)
Highlighted
L6 Presenter

You should take packet capture by filter both source and destinaion ip address.This will make sure you to see if there is a drop or no return.

Highlighted
L3 Networker

ok, thanks, ill try that

Highlighted
L5 Sessionator

You can find the definition for incomplete here: https://live.paloaltonetworks.com/docs/DOC-1549

As for the traffic, you can also look at the sessions to see if there are any packets from server to client. You can view this data from GUI through Monitor -> Traffic logs -> Click on the magnifying glass and look for packets sent and packets received. From cli you can do: show session all filter <interesting traffic> and show session id <id>. You can also look at the global counters to see if there are any drops.

You can alos refer to these documents for troubleshooting using counters & captures:

https://live.paloaltonetworks.com/docs/DOC-2542

https://live.paloaltonetworks.com/docs/DOC-2310

L6 Presenter

Incomplete denotes a lack of a tcp handshake. Syn packets allowed but never getting a syn-ack from the destination device. Perform client pcaps and span port on switch to help determine where those syn-acks are going if in fact the server is sending them.

Highlighted
L4 Transporter

You mention DR site. Are you facing asynchronous routing ?

Highlighted
L3 Networker

This can be due to not receiving enough packets for the decoder to identify the application.  Depending on what application it is, it may need more packets to identify.

Highlighted
L3 Networker

so i did a pcap.

seems the 'drop' log has a bunch of packets in it (outbound to server). So i guess they are being dropped on the way out on the initial connection attempt?

the transmit log is empty so i guess my packets are not even leaving the Firewall

however, if i test the fw rule from the command line everything seems fine (action allow).

test security-policy-match from ZONE source <My IP> to ZONE destination <Remote IP> protocol 6 destination-port 23

also, the rule as it stands right now is very loose.

@zarnia: Ill try looking at session data next,

@gafrol: I dont think there is a routing issue. The client LAN hooks up to the PA, then out through the IPSec VPN on the PA, so there not much room for different routes if any at all.

LAN -> PA/NAT -> IPSec VPN -> Remote Side

Highlighted
L6 Presenter

We're dropping packets? If you're good with flow basic, you can get granular information on why we're dropping? If not, please call into PAN Support as any debugging not done properly can impact services on or through the PAN device.

Highlighted
L5 Sessionator

Here are some good tips on debugging packet drops

1. Need to setup the filters for the traffic we are interested in. To do this, execute the following steps:

Navigate to Monitor--Packet Capture

Click 'Manage Filters'

Set Filter ID 1 to be the source IP and destination IP of traffic you feel is affected ( leave all other fields blank )

Set Filter ID 2 to be the exact inverse of what you did in step 3 (destination IP in source field, Source IP in destination field)

2. Setup up the captures

Create and name the file stage for a packet capture on all the stages (receive, transmit, firewall and drop)

3. Enable filters and captures

debug dataplane packet-diag set filter on

debug dataplane packet-diag set capture on

4. open 2 CLI windows

on 1 run the following command to look at the counter ( make sure it run this command once before running the traffic)

show counter global filter packet-filter yes delta yes

on the 2nd window run the following command to look at he sessions

show session all filter source <ip address> destination <ip address>

After your test has been done stop all the captures and filters and see if global counter show you anything why it is dropping the traffic or if you have getting pcap with drop stage.

This will help you narrow down the issue.

Let us know if this helps you resolve the issue.

Thanks

Numan

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!