Discussion of the Week: Application Incomplete

[reaper]

What does Application Incomplete mean? Reaper dives into the discussion topic and takes a closer look at how a session is established, packet flow, and what the firewall needs. See how this can impact the TCP handshake. Find answers on LIVEcommunity.

Earlier in the week, @kspg-itn posted a question asking why they were seeing Application Incomplete in one of their logs.

So what does this mean?

To understand how applications are determined, we need to take a deeper look at how a session is established and what the firewall needs to do during each step.

1. First, the client will initiate a connection by sending out a SYN packet.

This packet does not contain a lot of data, except for a source port and IP, destination port and IP, a handful of header flags, and the SYN flag.

To the firewall, this is the first step of the process:

• The source and destination IP addresses are matched against source and destination zones (by looking at the routing table).
• The security policy is checked for a match to the "5 tuple" (that is, does the source zone, source ip, destination zone, destination ip and destination port, match a security policy?

If a security policy is found to allow the packet through, the packet is forwarded (after NAT is applied) to the destination.

During this process, the firewall creates a session in the session table that will help future packets that belong to the same session flow through the firewall.

2. When the destination (server) receives the SYN packet, and it has a service listening on the port the client is connecting to, it will send a packet back with the ACK flag set to acknowledge receipt of the SYN packet. It will also send its own SYN packet to initiate bidirectional communication.

The firewall receives these packets and is able to match them to the existing session in the session table, because the ports, sequence numbers, and so on match the initial SYN packet. Because the session already exists, it does not need to check the security policy at this time and can simply apply NAT, if necessary, and forward the packets back to the client.

3. The client sends an ACK packet to confirm receipt of the server's SYN, then starts sending the first 'data' packets. These are packets that include an actual payload and could, for example, be an HTTP GET to receive a certain web page from a web server, or some other protocol.

4. The firewall will continue to pass packets back and forth, but meanwhile is inspecting all packets for identifiable information. If the server does reply with packets that resemble a web page, the App-ID engine can determine this session is actually a 'web-browsing' session and a few mechanisms are set in motion:

• The firewall policy is re-evaluated to verify if the detected application is allowed. At this point, the session could be dropped/rejected if the application is not allowed.
• The session is handed to the appropriate content engine to monitor the session to ensure it is behaving as expected and content is scanned for malicious packets.

What does this have to do with the application being incomplete?

During the lifetime of the session, the application may go through several stages before finally becoming what it actually is. When the first few packets are sent, there is no data except a the IP addresses and a port. In most cases, this is not enough to identify which application is being used.* The App-ID engine will classify this application as 'incomplete', as it is waiting for more packets for the handshake to complete.

Once the handshake is complete and some data packets have been passed around App-ID will most likely be able to match the payload against applications it knows and identify the session as such. If, however, App-ID is still not able to identify the packets, the session will be classified as  'unknown-tcp' (or 'unknown-udp') in which case this may be a homegrown application built by your development team, or a new app out on the interwebs. In this last case, you could consider creating your own custom app: Create a Custom Application

tl;dr Quick Answer

"Incomplete" means the TCP handshake was not successfully completed, either due to the packet never arriving, or arriving outside of the timeout window.

NOTE: In the case of DNS, for example, the first packet would already be identifiable as it has no handshake and the data for the look-up request are already included.

Recommended Reading

• Packet Flow Sequence in PAN-OS
• A book everyone should have in their bookshelf: TCP/IP Illustrated, Volume 1: The Protocols, by W. Richard Stevens and Kevin R. Fall
• Create a Custom Application

Useful Link for Customers

As we see this question quite often, a reminder that there is a page where our TAC team shares PAN-OS versions they deem most stable in most situations: Support PAN-OS Software Release Guidance.

Make sure to subscribe to the article, so you get notified whenever it is updated.

Feel free to leave comments or questions below or visit our discussion forum.

Reaper out!