How to efficiently block a large number of ip-addresses?

[mikand]

A discussion in a IRC-channel this evening was regarding the ongoing DDoS against wordpress installations all around the world and what to do in order to protect your webservers from the known bad ip addresses.

Using ACLs in for example a modern Cisco router seems to only be able to handle something like 1-10k ace's depending on masks being used etc.

Using iptables locally on the webservers seems to bail out after approx 30-40k lines with 10% cpu usage in kernel space for the linux kernel.

So what other countermeasures can be used?

A BGP based blackholing should be doable - however that only seems to block outgoing (returning) traffic from your network, it wont stop the incoming bad request (at least the syn will reach the server and then become hanging - while udp-traffic will be able to reach your server anyway). Unless im missing something here?

Another possible approach could be to enable HostnameLookups in your apache-server and use something like:

<Directory "/www/">

        Options                 None

        AllowOverride           None

<Limit GET POST>

        Order                   deny,allow

        Deny                    blacklisted.example.com

</Limit>

<LimitExcept GET POST>

        Order                   allow,deny

</LimitExcept>

</Directory>

and then in your DNS server make it authoritive for PTR records regarding the ip addresses you wish to block. When bad ip shows up your apache will ask the DNS for PTR-record of this host and if the answers from the DNS is "blacklisted.example.com" (or whatever you wish to call it) the apache will just drop the connection (perhaps with a http error 403 or such in return).

Which boils down to why I created this thread... what can be used if you have a PA device in place?

According to the datasheet the PA-5060 can use up to 40.000 security policies - but this is of course unique security policies.

A single security policy could perhaps be something like:

srczone: internet

dstzone: dmz

srcip: country(blacklist)

dstip: webserver

appid: any

service: any

action: deny

option: log on session end

or for that matter:

srczone: internet

dstzone: dmz

srcip: group(blacklist)

dstip: webserver

appid: any

service: any

action: deny

option: log on session end

So the question is really is it possible and in which way can one setup a custom country (or a custom address group) to hold 200k+ members (mostly /32 ip addresses)?

And as a sidenote - any efficient ways to keep this list easily up2date? There is for example methods to setup dynamic objects where the PA device will load a textfile from a webserver containing the ip addresses to act on as srcip (in this case) - or for that matter using the REST API to push (or withdraw) ip-addresses to blacklist. Will any of these methods work for a list that contains 200k+ ip addresses?

Or does any of you have other suggestions mainly from own experience? 🙂

[mharding]

With the dynamic address objects introduced in PAN-OS 5.0, it should be easier to accomplish. The hard part would be getting these bad IP addresses into a flat file for the XML-API. I was thinking about this the other day and trying to integrate with fail2ban. I just haven't had time to look into it. :smileylaugh:

[mikand]

Even if dynamic objects would solve the administrative hazzle they still doesnt seem to be the solution.

According to the linked technote: "Each dynamic address object can have 256 unique IP addresses associated with it."

This can of course be workedaround with some script-fu to create 782 dynamic address objects and put them into a single address group and put that into a single security policy... but then on the next page:

"

Each dynamic address object counts as one object towards the platform’s maximum objects threshold regardless of how

many IP addresses are registered to that object.  The maximum objects per platform and the maximum registered IP address

per platform (for 5.0.0) are detailed below:

Platform / Maximum Objects / Maximum Registered IP addresses

PA-5060 80,000 25,000

PA-5050 40,000 25,000

PA-5020 10,000 25,000

PA-4060 and PA-4050 40,000 5,000

PA-4020 10,000 5,000

PA-3050 10,000 5,000

PA-3020 5,000 5,000

PA-2050 10,000 1,000

PA-2020 5,000 1,000

PA-500 2,500 1,000

PA-200 2,500 1,000

VM-300 5,000 1,000

VM-200 2,500 1,000

VM-100 2,500 1,000

"

So back to square 1 😞

[scantwell]

I think  you should be looking at the Dynamic Block list, vs the Dynamic Address object. The Block list object allows you to put to a dynamically created block list, hosted on a webserver. Look up Spamhaus and see if theirs will work for you.

[mikand]

Isnt that what umphmharding already suggested?

And the limit regarding maximum number of ip addresses one can handle in a PA-5000 seems to be 25.000 which is far less than the +200.000 needed.

[HITSSEC]

Utilizing a list of +200,000 ip address may impracticle for a lot of us.  A couple of approaches may be helpful.  External block lists can be used. Entries in that list can be a single ip or up to a /16 if needed (utilizing subnet masks may cut down on the numbers).  The use of resource protection under QOS protection can be used to limit the number of connections you will allow to your wordpress servers from regions or the internet as a whole.  We have also used custom vulnerability signatures to stop external users from accessing specific directories and or files on our web servers.  The use of any or all of these techniques may reduce the risk to a more manageable level.

[mikand]

Thanks for the suggestions but the main problem remains - how to efficiently block shitloads of ip-addresses?

The maintenance of the address-list is fairly simple - create a sql-table and use the export function to dump the addresses in sorted order into a txt-file on your webserver when needed (and if you got some time create a web-gui to make it easier to search and handle the db).

And with the help of the dynamic block list will make the life easy for the admin - with the downside that not even the PA-5000 series can hold more than 25.000 addresses...

The problem is that there doesnt seem to exist any hardware today that can load these +200k addresses and filter them without any noticable performance-drop (for example iptables works up to approx 30-40k).

A fugly workaround might be to use a couple of 48 int cisco switches (given that it can do 1k ACEs per interface), setup 24 VLANs on each and then just a bunch of networkcables to connect the VLANs in serial with each other. You would need about 400 interfaces (200 VLANs) but you would solve the problem (or about 10x boxes running iptables)... which seems odd to me that in year 2013 the network hardware is still very limited in this area.

I mean the TCAMs used 1k ACE's in the 90s and they still seem to have the same limit today while computing overall is way faster and have way more memory today than back in the 90s :S

How is the geoip stuff handled within a PA device? Could that be used to put all these 200k addresses into a single country named "blacklist islands" and then just create a single security policy where you drop any traffic where srcip = country(blacklist islands)?

Or for that matter setup a security police which use FQDN (srcip = blacklist.example.com) with roundrobin (and the roundrobin is a list of 200k ip-adresses)? :smileysilly:

[scantwell]

Mikand, to answer your question... Dynamic Objects and Dynamic Block list are NOT the same thing. Dynamic Block list is https://live.paloaltonetworks.com/message/23654#23654 Dynamic objects (virtualization) is https://live.paloaltonetworks.com/docs/DOC-4121 Your answer/post is correct in referencing the limitations of the Block list functionality, but I wanted everyone on the post to know there is a difference, because the 2 were being intermingled. Thanks.

[dill]

TippingPoint IPS can do this using custom reputation entries.  Identify the attackers by Wordpress Admin login attempts (TP filter 12373) exceeding whatever threshold you set, use the sms responder to quarantine and add the offender to a custom reputation list (actions effective across all IPS enterprise wide).

As far as I recall, the N and NX series have no practical upper limit in IPv4.  (old E series, was capped at 10k)

[mharding]

Maybe we're all thinking about this the wrong way.

PAN-OS has DoS security profiles that work really well, after going through the work of tuning them.

There's an HTTP 401 Brute Force signature in the Threat-DB (https://threatvault.paloaltonetworks.com/Home/ThreatDetail/40031) that the threshold can be changed. ()

Just a thought.

[mikand]

Sorry about that...

But in this context it doesnt seem to matter if you use a dynamic address object or a dynamic block list - the limit is still 25.000 ip addresses on a PA-5000 box, isnt it?

[mikand]

Yes in this particular case using some kind of IPS signature to trigger the DoS-protection is probably the way to go.

But then what?

How many ip addresses can the PA dynamically block by the DoS-protection feature?

Doesnt it too have the same limit of max 25.000 entries on a PA-5000 series box?

Which brings me back to the original question - ignore the wordpress thingy. The use-case is that you need to block 200.000 /32's - how the hell are one supposed to do that nowdays without a major performance drop (and that is without putting 8 x PA-5000 boxes in a line :smileysilly:)?

[abelgard · Accepted solution]

What if you use BGP blackholing in addition to uRPF (IP spoof protection)?  I'm not sure if this is any more efficient than using an ACL on your router, but assuming your router has enough memory to hold all the /32 entries in its routing table, it may be worth a try.  Since the router's core function is routing packets, hopefully a route lookup for uRPF enforcement would be faster than an ACL check.  In theory, if your blackhole route doesn't point towards the internet, then the router should discard any packet from a blackholed IP when it arrives on your internet interface.

[HITSSEC]

We are dealing with the wordpress issue by blocking access to the wordpress login page from the internet. This is done with a simple custom vulnerability signature with default action allow (for inside users) and an exception of block (for outside users).  Content management must be done from the inside or if needed from the outside - they need to us a vpn connection.

[mikand]

Hmm yeah this seems to be the (currently) only proper and practically usuable way to deal with a situation where one would need to block plenty of ipaddresses.

It also has a name 🙂

RTBH - Remotely Triggered Black Holes

http://tools.ietf.org/html/rfc5635

http://tools.ietf.org/html/rfc6666

http://packetlife.net/blog/2010/aug/23/source-based-rtbh/

https://www.inex.ie/rtbh

http://www.netnod.se/files/download/453

You would most likely need to up the "maximum-prefix" in your BGP-settings like so (and use a device which can store and use +200.000 routes):

neighbor x.x.x.x maximum-prefix 250000

Which gives - how many routes (through BGP) can various PA-devices deal with?