Problems with XFF cleaning

[AndreasB]

Dear all,

we are getting more and more problems with the way PA handles the X-forwarded-for header.

It is very useful in getting the internal client IP in a proxy environment. OTOH, you need to clean it before it goes out so that you don't leak internal IP addresses.

This cleaning creates a problem.

PA only replaces the content of the header with blanks, so that it doesn't change the packet size.

With this approach we are getting more and more reports from users and external web site operators about us attacking them.

It looks like the empty XFF header triggers signatures on other IDS/IPS devices like:

"Apache HTTP Server X-Forwarded-For Denial-of-Service".

Is anybody else experiencing similar problems?

Regards,

  Andreas

[mikand]

A really ugly workaround would be to place a transparent squid proxy between PA and Internet to clean out any unwanted headers as described in: How to use or configure High Anonymous Proxy squid 2.6 stable21 release

A better approach would be to use a http-proxy which has the capability of keeping the srcip of the client when forwarding the traffic. This way no x-forwarded-for header is added and there is no need for such.

I know the Farist Firewall (Färist Firewall — Tutus) has this capability but also newer versions of squid as described in (look at my post from Jan 23, 2013 11:53 AM for urls).

Regarding manipulating the http stream through PA, can this cleaning be applied for the full row where the "X-Forwarded-For" exists? Or would the IDS/IPSes start to complain that the http header suddently has a blank newline? If so - what about cleaning out including the \n in the end?

I dont know if the ASICs/FPGAs which PA uses has the capability of altering the headers this way. Things that might break can be http-length (but thats for the http payload isnt it so changing the http-header shouldnt affect the http-length?) and perhaps ip-lengths for each packet (or in case this occurs in a fragmented packet for some odd reason and such).

I guess the best option currently is, unless you can get a Farist or a Squid as http-proxy, to file this as a feature request and if possible update this thread when you get a reply on this topic from your SE.

[gafrol]

Hello Andreas,

andreas.bischoff.cs schrieb:

It looks like the empty XFF header triggers signatures on other IDS/IPS devices like:
"Apache HTTP Server X-Forwarded-For Denial-of-Service".

Are you really sure this IPS event is related to how PA handles XFF ?

I see a lot of these IPS events on our customers IPS recently, and I doubt it's just because every company starts using PAN FW's stripping Proxy headers 🙂

cheers

Roland

[mikand]

True 🙂

It also seems to be related to multiple X-Forwarded-For headers in a single request:

National Vulnerability Database (NVD) National Vulnerability Database (CVE-2012-3526)

So either these IDS/IPSes are broken or the packets leaving your networks for Internet for some reason contains two (or more) X-Forwarded-For headers in the same request?

[AndreasB]

there is also a bug in the way PA handles such traffic:

The first request with the blanked out XFF header gets denied by the destination IPS. PA then resends the packet, this time with the header present.

May be that this retransmission triggers the signature.

[mikand]

Could you get a pcap of this behaviour?

Sounds somewhat odd that 1) the IDS/IPS sends a NACK to block the traffic (instead of just RST or just silently drop the traffic) but also 2) that the PA on the resend sends the original packet instead of the modified one?

If the 2) is verified then you should file this as a bug to PA/your SE.

[AndreasB]

At the moment this is handled as a bug, not as a feature request.

No ETA for a fix yet.

A possible fix is  to replace the whole line with blanks to keep the packet size. This would add a bunch of blanks to the preceding line. Should still be RFC compliant, but you never know if this will break other sites.

[AndreasB]

I can't post a pcap, but here the analysis from one of the sites which failed:

After analyzing this issue I think we finally found the problem.

Our IDS System was dropping your HTTP packet due to a malformed XFF header. Judging from your pcap file the XFF header in the first GET is empty, while in the retransmission packets the XFF value is correctly filled with an IP address. This behavior forces our IDS to block this connection attempt as it could lead to a possible Denial of Service attack.

We filed a bug report but apparently this is the expected behavior.

Getting the original bug fixed by completely blanking out the XFF header should fix this issue too.

[mikand]

So what about the thing I wrote in to configure your current http-proxy to keep the srcip (instead of using X-Forwarded-For) and then do the SNAT in the PA?

[AndreasB]

Currently impossible with two layers of proxies and a firewall inbetween. Don't ask me why

[mikand]

No, I meant replace (or reconfigure) your current http-proxy so the end result becomes:

Clients - HTTP-proxy - PA - Internet

The clients can then either go straight for the Internet IPs OR configure proxy-settings and go to the ip for the HTTP-proxy (depending on how you do today).

The HTTP-proxy will then do its magic but compared to from how it works today the srcip of the packet will be the client-ip instead of the HTTP-proxy interface ip towards PA.

The PA will do its magic regarding IPS, SSL-termination, URL-categories, AppID etc and before throwing the packet out on the Internet it will SNAT the srcip so the PA interface ip towards Internet will be used as srcip.

Things to do:

1) Change (or replace) the HTTP-proxy so keepsource=yes is being used on the ip header (that is client-ip is kept on the packets leaving the HTTP-proxy towards your PA).

2) Setup SNAT in the PA device.

3) Configure a static route for the PA device so it can find its way back to the clients.

Edit:

4) Disable X-Forwarded-For in your HTTP-proxy 🙂

[gafrol]

Just a side note regarding the IPS Event (http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-3526)

This filter detects an attempt to exploit a denial-of-service vulnerability in vulnerable installations of Apache HTTP Server.  An unauthenticated remote attacker can exploit this vulnerability by sending HTTP requests with malicious X-Forwarded-For headers to a target which could result in a denial-of-service condition.

Note: Enabling this filter will likely cause false positives if used in conjunction with anything other than Apache Server mod_rpaf 0.5. (taken from HP Tipping Point IPS)

mod_rpaf 0.6 is the current version and not vulnerable to that attack. Usually this Event turns out to be a false positive just because there is no apache reverse proxy in place and/or the mod_rpaf is not being used at all.

The art of IPS tuning

[gafrol]

Andreas and I did a quick test with our IPS and indeed the way how PAN strips the XFF content seems to trigger the IPS Event.

When I look at the pcap from the IPS event I see:

I believe this can bee seen as proof that the PAN XFF handling fires these IPS Events. PAN can you fix it ?

[mikand]

Would it be possible to use a regexp in PA to alter the header if the header cannot be deleted all together?

Im thinking if the request is:

X-Forwarded-For: 10.123.123.123

then alter it so it becomes

X-Forwarded-For: 10.0.0.1

or for that matter putting the ip of the egress interface there?

That is "NATing" the X-Forwarded-For (when passing through PA) so it will always display the same ip no matter how the original request looked like (given that there existed a X-Forwarded-For header in the original request)?

Or would this break other things on serverside?

[Anon1]

With PAN-OS 6.1.5 a new configuration option has been added to overwrite xff header with IP address 1.1.1.1 instead of blank.

https://www.paloaltonetworks.com/documentation/61/pan-os/pan-os-release-notes/pan-os-6-1-5-addressed...

Fixed an issue where, with Strip X-Forwarded-For (XFF) enabled ( Device > Setup > Content-ID), the firewall stripped the XFF IP address and inserted spaces. With this fix, you can execute the "debug dataplane set x-fwd-for-enhanced on" CLI command to configure the firewall to replace the XFF IP address with 1.1.1.1instead of spaces.