Easily fooling App ID?

[tweaked2]

Recently I've seen some information posted on a forum and I was hoping someone could speak to this.  Here is a quote:

"I work as a security auditor and I've come across them a few times. The "application" is http (web) which is allowed from the client segment to the external one. The backdoor software mimics a regular web session, as per a user would do.

The PA initially blocked the session, changing the User-agent string from Metapreter to a valid browser, be it Internet explorer, Mozilla, etc allowed the software to call home and establish a remote shell.

Specifying ACLs based on applications is NOT how you do it, you create an IP-ruleset and then, if you feel like it, apply applications to that ruleset. PA firewalls only look for the header information in connection attempts, that's another way of bypassing them. Start the session via http, change over to ssh or anything you feel like it because it's already added to the state table as a valid connection."

They also pointed me to this video for how NGFW's work and their flaws.  DEFCON 19: Network Application Firewalls: Exploits and Defense - YouTube

Although the video is speaking more towards a layer 7 only device vs a NGFW.

He also said that the PA he encountered and compromised was installed by a PA partner using best practices.  Logic would state if a PA were this easy to compromise we would have heard it in the news by now. 

Any input would be appreciated. 

[mikand]

Sounds like a FUD race from a competitor which doesnt like the competition from PA 🙂

Here is the response I got from "my" SE regarding a similar statement and the defcon 19 video presented by a Juniper guy:

"

Hi Folks,

It is unfortunate that i need to send this out but i have received several

questions form partners and customers now to comment on some of the

messaging seen from Juniper on Application firewalling and its short

comings, and we feel that we need to point out some things in the

messaging that are not quite correct when applied to our technology.

Otherwise we love that everyone is discussin nextgen firewalls to build

the market.

I provide this so you can provide your customers with good information, it

also gives you a good summary on how AppID actually handles special and

unusual cases.

Let us know if you have any followup questions.

Claim: Application firewalling does not replace the need for IPS.

Absolutely true. There is no question that IPS technology is a

requirement. App-ID is not intended to replace IPS or even reduce the need

for it. This is the reason we have invested in building a best-in-class

IPS functionality into the device. This is also why IPS is a critical

piece of the next-gen firewall requirements.

Claim: Application cache causes traffic to be misidentified.

Not true. App-ID cache delivers performance gains without sacrificing

accuracy. Application and protocol decoders validate the traffic on

sessions identified using the cache, and if the validation checks fail,

the cache entry is removed. The session is then switched out of the

decoder and identified as unknown. Also, at short periodic intervals, new

sessions are inspected by the App-ID engine regardless of the cache hit.

This handles cases where the destination/port may have been reused for a

different application server. We repeated the test described in their

claim by populating the App-ID cache with an entry for web-browsing by

sending multiple HTTP sessions on the same destination/port. The SMTP

traffic in a following session on the same destination/port causes a cache

hit, is inspected by the HTTP decoder, fails validation checks, is

switched out of the decoder, and reported as unknown. The cache entry is

invalidated as part of this and the next SMTP session will be identified

as SMTP.

Claim: Does not inspect both Client-to-server and Server-to-client

directions of traffic.

Not true. Traffic is inspected in both directions. We repeated the test

described in their claim, sending an HTTP GET request followed by a FTP

server response on the same session. The session is initially identified

as web-browsing but switches to unknown-tcp when inspecting the FTP server

response, since the traffic is neither HTTP nor FTP.

Claim: Can evade Application firewall using client-server collusion by

starting the connection as a permitted application and switching to

another.

Not a genuine or valid test. The scheme devised in the test assumes both

the client and server are already under the control of the attacker. We

don¹t know of any real-world clients or servers that can talk HTTP

initially and then switch to SMTP. In any case, App-ID has coverage for

many evasive real-world tunneling applications and we continue to add

coverage for more as we discover them.

Claim: Some applications can only be identified on specific ports.

Partially true. The application example given was DNS. The identification

scheme for DNS includes a check for port 53 since we don¹t expect any real

world DNS service to be running on any other port and since DNS traffic is

often very short and difficult to reliably identify by signature alone.

Non-DNS traffic sent to port 53, will be validated in the DNS decoder to

ensure it is really DNS. If the validation checks fail, the session is

switched out of the DNS decoder and identified as unknown.

Claim: Does not inspect traffic at Layer 7 when the Application can¹t be

identified.

Partially true. Traffic that does not match any of the applications that

we have coverage for will be reported as unknown and eventually stop being

scanned for threats. The best practice for dealing with unknown traffic is

to deny it in security policy. Customers using an application that we

don¹t currently have coverage for, can submit a request for adding

coverage for it in one of our weekly content updates, or create a custom

App-ID on their own. Allowing unknown applications is a risk that other

firewalls or IPS products cannot prevent.

"

In my opinion you cant entirely rely on appid - simply because appid triggers on various behaviour and strings like a signature (which is also why appid's are updated every now and then), thats why I recommend people to never use service:any in an allow rule but at least use service:application-default unless you can specify which port(s) you want to use like service:TCP443.

Meaning when you build your security rules you start to configure the PA as a regular SPI firewall (srcip, dstip, srcport, dstport), THEN you enable appid for each security rule (where needed) and perhaps userid and urlcategories (and AV, IPS, File etc) and so on. In reality you do this line by line but to explain the flow of how (at least I :P) thinks when I setup rules in a PA.

In order to see how application identification works in real life you can use this simple test:

Setup a rule which will allow "appid:web-browsing" (basically any HTTP based traffic) along with "service:any" (yeah I know I said you should NEVER EVER use service:any in an allow rule but this is just an example 🙂 along with a http server who listens at TCP81.

When the client sends a request such as "a b c" (instead of "GET / HTTP/1.0" which is more like a true HTTP request) the packet is allowed through to the server. Not until the server replies with "HTTP ERROR 400/Bad Request" the AppID engine will know that this flow/session is supposed to be a HTTP communication but "a" is not a valid HTTP request and the response will never reach the client.

Workaround for above (if you dont want the request to hit the server) is to deny "appid:unknown" but also create a custom AppID (application signature) or ThreatID (IPS signature) which demands that the "HTTP-Request-Method" should be HEAD, GET or POST (or which HTTP request methods you wish to allow). Another method (along with previously mentioned methods) is to use "service:application-default" or if possible "service:TCP80" (given that your server normally would listen to TPC80 and the stuff at TCP81 was some other service). You can also put a proxybased firewall inline with the NGFW device to further inspect the contents of the traffic (depends of course on how the proxyfirewall is being setup but still). There are some options which in combination can be a sufficient "workaround" to handle this testcase security wise.

[ACieszkowski]

"Claim: Can evade Application firewall using client-server collusion by

starting the connection as a permitted application and switching to

another."

Blog entry brought to my attention describes this technique: http://www.what2code.net/?p=100

[mikand]

I dont think this blog example shows a protocol switching technique but rather a protocol tunneling technique.

1) Of course this will work becuase the one who did the blog post use a "any any allow" as seen in http://www.what2code.net/wp-content/uploads/2012/10/11.png

2) Also no IPS signatures are enabled (not that they might stop this example but still).

3) As seen in http://www.what2code.net/wp-content/uploads/2012/10/61.png the one who did the blog post is using a custom (http) tunnel application (look at the name of the window for example: "Form1"). Even if PA currently covers many evading and tunneling techniques there will still be others available (and new will be invented over time - thats why you need to have your appid-db up2date).

4) Which bring us back to the first note above - you DONT do "any any allow" as default last rule when firewalling because then you will end up in situations like this one (blacklisting is always hard and time consuming). What you do in reality is default deny and only allow the traffic you wish to allow (whitelisting is easier than blacklisting). Actually if you remove that last "any any allow" rule which this person added the PA device would drop this traffic on its own (because PA uses what the marketing people calls "positive enforcement").

5) Also PA will only detect one appid per session at a time. Meaning a single session can only have one appid at a moment. So in this case since the (http) tunnel being used isnt detected as its own appid the outer protocol is http and obviously being detected as "web-browsing" (web-browsing can be described as "http traffic which doesnt match any other already known appid that is http-based").

So to sum it up:

1) User error (its a really bad idea to have default allow).

Which also gives why it isnt enough to just put a firewall in your network and believe you are done. You still need basic infrastructural/architectural design that WHEN (and not IF) shit hits the fan the (for example) infection can still be mitigated (for example by not allowing clients to speak to each other etc).

2) Make a tcpdump and send it to PaloAlto so they can create an appid for this tunneling technique.

[mharding]

I concur.

I bet we'll see an update to the HTTP-Tunnel app in the coming weeks.

[ACieszkowski]

I can not fully agree with you.

In my opinion author is using protocol switching, not tunneling. Starts genuine HTTP session and then starts SSH in the same session (more in post: http://www.what2code.net/?p=18), particularly screenshots: http://www.what2code.net/wp-content/uploads/2012/07/3.png and http://www.what2code.net/wp-content/uploads/2012/07/4.png). If using tunneling he would be, lets say, pushing SSH data in responses to HTTP GET, or in <body> or similar. But classification of the method is not the issue here.

1&4) Yes and no. Yes, it is a bad security policy. No, it is not the case why this method works. If we replace "any any allow" with "any any drop" but add "any web-browsing allow" above, it would have ended the same. Blocking web-browsing for users would be hard, would it not?

3&5) Custom application is the one thing, other is that PA detects only one AppID in session as it starts. It does not verify its first match while session lasts. That behavior (flaw?) is used to cheat App-ID. Send HTTP, get HTTP. App-ID identifies the application as "web-browsing" (correctly). After that your are free from the ties. Drop any application into that session and PA will be blind to it.

PAN could deal with that introducing validating session's application at random periods of time (I suspect the performance impact would be significant).

[mikand]

Im always a bit sceptic to closed source "disclosures" because there is often one or more "tweaks" not being mentioned that makes the particular example to work:

"

I will not publish the source code, the server script has been written in Perl, and the client one in C#.

"

I mean in theory this person could just have manually disabled L7 detection in the PA which of course basically disables appid ability to track changes in a single session.

In this blog I also later found out about two drawings explaining the overall design of this evasion however no details are really being showed:

http://www.what2code.net/wp-content/uploads/2012/07/serverforwarder.png

http://www.what2code.net/wp-content/uploads/2012/07/clientforwarder.png

Given that the two pics you linked to:

http://www.what2code.net/wp-content/uploads/2012/07/3.png

http://www.what2code.net/wp-content/uploads/2012/07/4.png

are in the same session we can still see that there are missing some data between them two.

It seems like the server answers with a real gif-picture but in the end of the packet injects some SSH server code. To which the client then replies with its own version and a SSH-handshake is being performed.

So questions which hopefully PA can answer:

1) Does this show a bug in the http-decoder/appid-engine?

2) Would enabling IPS react to this sudden "SSH-2.0-OpenSSH_5.5p1" in the middle of the http-transaction?

Regarding your comment to 1&4) Well the case was "we want to block ssh" according to the post, not "we want to block ssh and allow everything else including web-browsing". Because if you alter the security policy to allow web-browsing then why not at the same time alter the policies to also enable the builtin IPS? Or for that matter create your own IPS signature to block traffic where "SSH-2.0-OpenSSH_" is being seen?

I also wonder if this really can be fixed. I mean the server reply which is a binary file such as a gif could of course contain a string such as "SSH-2.0-OpenSSH_", so PA cant block traffic based on that (due to high probability of false positives). However the client response along with the ssh-handshake should be detectable - which is also why I would like to see the same test with the IPS enabled (because the IPS acts on allowed traffic).

I think its good that bugs (with the disclaimer that this is a bug) are found and fixed - but the blogpost is slightly more of a tabloid news paper style which leaves out several details not to mention that almost noone would use a "any any allow" as a default last rule in a firewall. I mean the specific finding cannot be verified by anyone else since the source code is not being published (which when you come to science is like the major issue - whatever you find shall be able to be reproduced by somebody else).

And regarding 3&5) to my knowledge various protocols have various ways of identifying what kind of application it is. If the session is offloaded to the http-decoder and the http-decoder fails to detect that this is no longer the same http-session containing http-data then there is a bug in the http-decoder.

According to the answer from "my" sales engineer the PA do verify appid's even in already verified sessions from time to time.

Another sidenote... by default PA logs on session end... during debugs you often want to enable on session start to see how the appid's is being switched such as unknown etc. I wonder if there would be the same result in the log if both on session start AND on session end would be enabled?

A possibility could be that the appid when tcp-session is closed is web-browsing, but until that there were perhaps both unknown and ssh (but not shown and due to "any any allow" they are allowed)?

Oh and regarding "locking web-browsing for users would be hard, would it not?" - not at all, use a terminal server solution where the terminal servers are locked down and voila, the users have no direct communication to the internet from their client stations and you as the admin have control of the client environment (as in the terminal servers).

[ACieszkowski]

I understand your skepticism, but we can not approach this with "source code or it didn't happen" 😉 And the idea of the post was rather "how to bypass App-ID" - rules and they order are not the case here.

Please check: http://www.what2code.net/?p=100#comments - author says that he enabled IPS without difference in the PAN-OS behavior. Moreover he has opened case for the bug.

I was informed by my SE that PAN TAC is aware of the problem and working on the solution.

Can this be fixed? Yes: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... Question is: when PAN will fix this?

Released patch from Check Point confirms that this method works, most probably also on PAN (TAC is working on the issue, right?).

I wonder: is this http only issue? Or is this bigger: SMTP, FTP, other protocols?

[mikand]

The reply I received from "my" sales engineer was that this is a confirmed bug in the http decoder and a fix will be released shortly.

My sceptism comes from years in this business when you will learn that things are not always as they first showed up as (same goes with PA for that matter 😉

Regarding this specific blog post, yes (and I already said this) its great to point out bugs and other misbehaviours from equipment - but an even greater lession (which the blogposter seems to miss) is how NOT to setup security policies (in production).

There will most likely be more of similar findings in future and having an "any any allow" as last rule will always make you vulnerabile to this and similar bugs.

Another lession which should be learned is of course that you cannot trust a single box. You must have a design to when shit hits the fan so the spreading will be limited or can even easily be quarantined and mitigated. Unfortunately thats not how most corporate networks looks like today where clients can (in most cases) without problems reach each other and the network is basically flat (with hopefully at least the servers segmented away - but then the servers can reach each other instead and bad things can continue to happen on your inside).

[mikand]

Can someone from PA confirm that this bug has already been dealt with?

2012-10-24 02:46

Version 335 Content Release Notes

Application and Threat Content Release Notes

Version 335

<snip>

Modified Decoders (5)

      Name

      freegate

      http

      gtalk-file-transfer

      ssl

      sccp

[mikand]

Got an answer from "my" sales engineer that the fix for the above case will be available in the Version 336 Content Release.