Why do "incomplete" sessions show as "allowed"

[darren_g]

Hi.

I've got some pretty specific firewall rules for machine in our DMZ, and I noticed some intriguing log entries while checking into an (unrelated) issue today.

I get a log entry which reads like this

02/27 11:42:30      end     outside     DMZ     <source_ip>     <destination_ip>     1433      incomplete     allow     <rule_name>     152

(I've obfuscated the IP addresses etc).

Why doesn't this show as a "deny" when the application doesn't match one on the permitted list for this rule? Even the port doesn't match one covered by any of the allowed applications in the rule.

It's not effecting anything, I'm just curious.

[jvalentine]

Incomplete either means that the 3-way TCP handshake didn't fully complete, or it completed without any additional data packets.  The App-ID engine identifies the actual application based on the actual data traffic after the 3-way TCP handshake.  If there was a partial handshake, or a complete handshake without any additional data traffic, the application will be marked as incomplete. 

[darren_g]

Yes, I understand why they show as "incomplete" - what I want to know is why they are logged as ALLOWED by the firewall, not logged as dropped. Surely, the logic would be "if it matches, allow, if it doesn't match, or doesn't complete, then drop"?

[mbutt]

Hi,

When the session is established and packets come in to the firewall.

Initially the traffic has to be leaked to make the determination of the application.

It is allowed on the most open security rule from top to bottom and left to right.

If the traffic is incomplete or insufficient traffic, it means the determination of the application could not be made or the tcp handshake did not complete.

Since the traffic was initially leaked to make the determination for the application and no further processing happened on it since it was allowed.

It shows up as allowed in the traffic logs.

Hopefully this explains it.

Thank you

[darren_g]

>Since the traffic was initially leaked to make the determination for the application and no further processing happened on it since it was allowed.

> It shows up as allowed in the traffic logs.

But the actual traffic *is* dropped, yes? I'd hate to think some of this stuff actually leaks to the servers inside the DMZ!

[jvalentine]

Let's look at this from two different perspectives:

The "old-school" port-based firewall.  You'd have a rule that says "permit from any to dmz_server tcp/1433".  If someone sends a tcp/syn on 1433 and stops there, would this firewall let it through?  Yes.  If someone sends a syn, gets a syn/ack back, and then sends an ack, does that get through?  Yes.  What about after the handshake and then they send MS-SQL data to the server, does that get through?  Yep.  What about telnet or ssh, does that get through?  Yes.  (of course, that assumes there's a telnet or ssh daemon listening on that port) - but the firewall is clueless at this point.  As long as it saw a tcp handshake on tcp/1433, it will let ANY application through that rule. 

Now let's look at this from a Palo Alto Networks firewall perspective.  You have a rule that says "permit from any to dmz_server ms-sql on tcp/1433".  Now, if someone sends a tcp/syn on 1433, that gets through (just like above).  What about syn, syn/ack, ack?  Yes, that gets through as well.  What about if, after the handshake, someone sends MS-SQL data to the server, does that get through?  Yes - and if they do, it will be logged as ms-sql.  What about telnet, or ssh, does that get through?  NO, it won't.  That's where the big difference is. 

I wouldn't call it "leaking" data.  It allows a handshake on that port, and then validates the application after the handshake takes place.  Unfortunately, there is no way to know what the application is until after the tcp handshake happens.  What you're seeing is tcp handshaking that is allowed under that rule.  The firewall didn't block it, but it would have validated the application and blocked any app that doesn't match the security policy.     

[ericgearhart · Accepted solution]

Just out of curiosity, what does your 'Service' column look like in your rule base? Is it set to 'application-default'?

I'm curious about what the answer to the "leak" question is as well

[ericgearhart]

Doesn't that kind of imply that if what you're saying is true, a simple SYN scan of a PAN firewall's external IP address would show hundreds of ports open?

That's assuming the 'Service' column in the firewall policy is set to any I suppose, not 'application-default' or specifically 'tcp-1433' (using your example of MS SQL)

[darren_g]

> I wouldn't call it "leaking" data.  It allows a handshake on that port, and then validates the application after the handshake takes place.

But the port upon which the handshaking is taking place is *not* in the allowed list. It's not an issue is the app identification occurring after the handshake - the initial (SYN, I assume) is being logged as "allowed" on a port which is *NOT* in the list of ports covered by the "allow" list.

In the example I originally quoted, the port identified is 1433 - MS-SQL - but the allow list for the rule concerned consists of

web-browsing (tcp/80)

subversion (tcp/3690, udp/3690)

webdav (tcp/443, tcp/80)

There is no allow for tcp/1433 which can possible be allowed under those rules.

[darren_g]

> Just out of curiosity, what does your 'Service' column look like in your rule base? Is it set to 'application-default'?

Damn, you're right, and I'm a butthead.

It is set to "any", Because I'm a bloody idiot and missed changing that when i set up this (and a couple of other) rules.

It'll shortly be set to "application default" and, I hope, resolve the issue.

Cheers!

[ericgearhart]

No big deal man it's an easy mistake to make believe me! When I first started working at the company I'm now with I was new to PAN, and the service column was weird and mysterious to me.

I'm almost of the opinion that there should be some kind of commit warning if the service column is left at 'any' for any rules in the policies tab that are bound to the 'untrust' zone. That's just my own humble opinion though.

[darren_g]

Thing is, I'm not "new" to PA. I had (have, I suppose) a PA certification in the V3 OS, and this is now the third place I've successfully pushed the installation of the PA firewalls into (yeah, I kinda like them compared to other stuff, how did you guess? :-)) - I just plain bloody *forgot*, and then asked a stupid question.

Of course the firewall was trying to identify the application before dropping on non-standard ports - because I didn't tell it not to!

I'm going to go hide my face in shame now.

[jvalentine]

Yes, if you create a rule that says "permit any app on any port" or "permit specific app on any port" the you could port-scan a machine from one end to the other. That's why you never, never, ever use "any" service on inbound permit rules.

[darren_g]

OK - how do you deal with application over-rides?

(I know, I know, they're a case of baby did a bad, bad thing, but I need one).

If I set "application default" on those, will they break anything else? In effect, an application override just looks at the port for the specified designation, right? So "application default" should be fine?

[mikand]

If I remember correctly the PA will do a precheck on the packet before it start to crunch it through the "singlepass" engine.

This precheck (described in a pdf/picture on this site) will check for the ip-header stuff like srcip, dstip, srcport, dstport. Which gives if you for example only have one rule setup and thats something like "appid:web-browsing, service:TCP80" then if the packet arriving has a dstport of something other than TCP80 then this packet will be dropped (and logged if you have a rule in the end which will log dropped packets, the builtin hidden last rule is just a drop without logging).

Thats why you should avoid to use "service:any" and always use at least "application-default" or even better define exactly which ports you allow.

The next step (when this "singlepass" kicks in) is to try to detect what kind of application this packet actually is, send it through a protocol-decoder and if thats successful it will do other tests aswell on the packet before finally (hopefully) give a final verdict of letting this packet pass through (or drop the session completely).

This gives that if your rule instead was "service:any" the precheck would always fail (given the combo of srcip + dstip is valid aswell) and the PA must let through enough of packets before the particular application can be identified.

Some applications can be identified on the first packet (after the tcp handshake if we talk about tcp applications) while others need a few more packets before the verdict can definately say "this packet is not the allowed appid".

You can see this yourself in action if you setup a rule like "appid:web-browsing, service:any" and put a webserver to listen at lets say TCP81.

Your tcp-handshake will complete (since its obviously part of web-browsing) and you will also be able to send a packet through towards the webserver - lets say a packet such as:

a b c

that is three letters with whitespace in between.

As you can see "a" is not a valid http method - but PA obviously doesnt know this (yet)...

not until the server responds with "HTTP ERROR 400/Bad Request" it seems that PA is 100% sure that this flow has nothing to do with web-browsing and will now drop the flow (and log it if you have such rule in the end).

I have no idea why PA will let through such requests (or if this particular case has been taken care of by an appid update) but a workaround is to create a custom appid which is based on web-browsing but with the addition of "http-method" (or whatever its called when you create custom appids) needs to be GET, HEAD or POST (or whatever http methods you like to let through). Now, with the custom appid, even the request would be dropped.

There are also other corner cases such as if you wish to block skype. In order to block skype you must first allow skype-probe. Which gives that the more you go into "applications" the more leakage will occur.

Personally I recommend to first configure your PA device as you would with an older SPI based device - look at srczone, dstzone, srcip, dstip and service. THEN add which application is supposed to get detected by this flow. And in some cases also add a url-filter (if dealing with browsing). And of course the rest of a proper IPS profile, AV, SSL-termination (to inspect https sessions) etc.

Edit: Here is the document I had in mind:

There is also an "executive summary" edition of the flows described on page 4 (anyone else remembers what that document is called? its like light green boxes describing the same flow but with less technical details).

Edit2: Here is that "executive summary" (well sort of) I had in mind: http://media.paloaltonetworks.com/documents/techbrief-app-id.pdf