Problems with XFF cleaning

[AndreasB]

Dear all,

we are getting more and more problems with the way PA handles the X-forwarded-for header.

It is very useful in getting the internal client IP in a proxy environment. OTOH, you need to clean it before it goes out so that you don't leak internal IP addresses.

This cleaning creates a problem.

PA only replaces the content of the header with blanks, so that it doesn't change the packet size.

With this approach we are getting more and more reports from users and external web site operators about us attacking them.

It looks like the empty XFF header triggers signatures on other IDS/IPS devices like:

"Apache HTTP Server X-Forwarded-For Denial-of-Service".

Is anybody else experiencing similar problems?

Regards,

  Andreas

[mikand]

A really ugly workaround would be to place a transparent squid proxy between PA and Internet to clean out any unwanted headers as described in: How to use or configure High Anonymous Proxy squid 2.6 stable21 release

A better approach would be to use a http-proxy which has the capability of keeping the srcip of the client when forwarding the traffic. This way no x-forwarded-for header is added and there is no need for such.

I know the Farist Firewall (Färist Firewall — Tutus) has this capability but also newer versions of squid as described in (look at my post from Jan 23, 2013 11:53 AM for urls).

Regarding manipulating the http stream through PA, can this cleaning be applied for the full row where the "X-Forwarded-For" exists? Or would the IDS/IPSes start to complain that the http header suddently has a blank newline? If so - what about cleaning out including the \n in the end?

I dont know if the ASICs/FPGAs which PA uses has the capability of altering the headers this way. Things that might break can be http-length (but thats for the http payload isnt it so changing the http-header shouldnt affect the http-length?) and perhaps ip-lengths for each packet (or in case this occurs in a fragmented packet for some odd reason and such).

I guess the best option currently is, unless you can get a Farist or a Squid as http-proxy, to file this as a feature request and if possible update this thread when you get a reply on this topic from your SE.

[gafrol]

Hello Andreas,

andreas.bischoff.cs schrieb:

It looks like the empty XFF header triggers signatures on other IDS/IPS devices like:
"Apache HTTP Server X-Forwarded-For Denial-of-Service".

Are you really sure this IPS event is related to how PA handles XFF ?

I see a lot of these IPS events on our customers IPS recently, and I doubt it's just because every company starts using PAN FW's stripping Proxy headers 🙂

cheers

Roland

[mikand]

True 🙂

It also seems to be related to multiple X-Forwarded-For headers in a single request:

National Vulnerability Database (NVD) National Vulnerability Database (CVE-2012-3526)

So either these IDS/IPSes are broken or the packets leaving your networks for Internet for some reason contains two (or more) X-Forwarded-For headers in the same request?

[AndreasB]

there is also a bug in the way PA handles such traffic:

The first request with the blanked out XFF header gets denied by the destination IPS. PA then resends the packet, this time with the header present.

May be that this retransmission triggers the signature.

[mikand]

Could you get a pcap of this behaviour?

Sounds somewhat odd that 1) the IDS/IPS sends a NACK to block the traffic (instead of just RST or just silently drop the traffic) but also 2) that the PA on the resend sends the original packet instead of the modified one?

If the 2) is verified then you should file this as a bug to PA/your SE.

[AndreasB]

At the moment this is handled as a bug, not as a feature request.

No ETA for a fix yet.

A possible fix is  to replace the whole line with blanks to keep the packet size. This would add a bunch of blanks to the preceding line. Should still be RFC compliant, but you never know if this will break other sites.

[AndreasB]

I can't post a pcap, but here the analysis from one of the sites which failed:

After analyzing this issue I think we finally found the problem.

Our IDS System was dropping your HTTP packet due to a malformed XFF header. Judging from your pcap file the XFF header in the first GET is empty, while in the retransmission packets the XFF value is correctly filled with an IP address. This behavior forces our IDS to block this connection attempt as it could lead to a possible Denial of Service attack.

We filed a bug report but apparently this is the expected behavior.

Getting the original bug fixed by completely blanking out the XFF header should fix this issue too.

[mikand]

So what about the thing I wrote in to configure your current http-proxy to keep the srcip (instead of using X-Forwarded-For) and then do the SNAT in the PA?

[AndreasB]

Currently impossible with two layers of proxies and a firewall inbetween. Don't ask me why