Denying the DDoS

Latest response

With the recent uptick in Distributed Denial of  Service (DDoS) attacks, we're often asked what steps can be taken from an OS-perspective once communication gets past firewall and other filtering devices out on your network's edge.  It's a sad reality of our technological lives today that attacks like these are here to stay.  So what do you do when dozens, or hundreds, or thousands of bots are poking and probing at you?  While there is no magic pill that can defend against every attack, the best defense you can field is a layered one that helps compensate for gaps within any one technology.

Some of the suggestions I'll give are not Red Hat-supported solutions, but have really proved themselves out in the Community, so are valuable to note.   (insert lawyer-speak "Some of the  software/modules mentioned here are not officially shipped or supported by Red Hat. According to Red Hat's "production scope of coverage" (Defined at https://access.redhat.com/support/offerings/production/soc.html), "Implementation and development of security rules and policies" is not supported. suggestions given in this case are purely based on knowledge and experience and are given for reference purpose only." )  Before implementing ANY new change to your environment please ensure you've thoroughly read and understand all instructions/documentation and have tested out your new configuration in a non-production environment to measure the impact it will have.

First and foremost, if you don't need a service or a port exposed to the world, turn it off.  Period.  Only turn on what is absolutely necessary to conduct your business, anything else is inviting unneeded risk.  The same thing goes with Access Controls, if people don't need to read it, remove permissions to it.  Harden your server and give it the smallest possible footprint of exposure.

Now sometimes you absolutely can not *not* have something exposed.  In those cases, iptables can be your bestest-buddy ever.  iptables can quickly and effectively used to block traffic by port, protocol, ip, etc.

More information about iptables can be found in the Red Hat Linux Security Guide:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Security_Guide/sect-Security_Guide-Firewalls.html

Be sure to test any changes prior to executing.  Another handy thing to do is once you've put a solution in place is to review any logs to make sure you've stopped the bad traffic and are still allowing the good.  

   How do I setup logging in the iptables firewall?
     https://access.redhat.com/knowledge/solutions/6249

For those of you out there that are Apache-enthusiasts, there are two modules that can be leveraged to help manage (and stop) unwanted traffic.

mod_qs is a great lightweight tool that can quickly and easily be integrated with your webserver.  It isn't as feature-rich as others, but it certainly can do the trick.

A more advanced option might be to leverage mod_security.  This is a robust open-source web application firewall.  It's much more involved than mod_qs ad will require a higher-level of expertise to really get the best protection, but it does a great job.

     http://www.modsecurity.org/
     http://opensource.adnovum.ch/mod_qos/

All of this might not be enough, and you have to work with your Edge provider and start to blacklist attackers.  That's where tools like Project Artillery or (D)DOS-DEFLATE might come in handy.  Again, your mileage may vary and changes might need to be made to make non-Red Hat supplied packages work on your RHEL systems.

     https://www.trustedsec.com/downloads/artillery/
     http://deflate.medialayer.com/

So what would you add to help someone faced with a DDoS attack?  Any tips or tools you've found useful in defending your castle?

Responses

This is a great writeup! I would say though, if you are concerned that you may be under a DDoS, your first step should be to contact your edge provider so they can be teed up in case they are needed, and possibly do some proactive work on your behalf (as it would also prevent collateral damage to other customers on your network).

That being said, I'd be curious to see some other feedback on how folks leverage the technology to mitigate DDoS attacks.

For sure.  Anytime someone's under these types of attacks it's going to be a bigger effort than just "the server team".  Edge providers, DNS admins, network route/switch/firewall, web teams.  Depending on the size and scope of the attack it's likely to be an "all hands" event.

SYN cookies protection can be used when the system is under a SYN flood attack and source IP addresses of SYN packets are also forged (a SYN spoofing attack).

# echo 1 > /proc/sys/net/ipv4/tcp_syncookies 

 

Fortunately, many applications offer built-in rate limiting functionality.

For applications that are launched through an inetd type of master-service, xinited offers rate-limiting options for the network services it manages.

For applications that neither have built-in rate-limiting nor can be managed through xinetd, the iptables "recent" module can be good for clamping down on too many accesses from either individual IPs or networks.

I'm working for an ISP, and some our customers has lately been part of DDoS attacks where the customers CPE (or other equipment in the customers home) has been proxying large numbers of spoofed DNS queries to our DNS-servers. What we see on the DNS-servers is large numbers of queries for "ANY ." and "RRSIG ." -- typically DNS amplification attacks where small queries generates large replies.

To protect against this we've started by implementing rate limiting for these two queries in iptables. The packets are identifiable using the "u32" iptables module:

-------------------------

:BUCKETLIST - [0:0]

-I INPUT -p udp -m udp --dport 53 -m u32 --u32 0x28=0xff00 -m comment --comment "Limit ANY . queries" -j BUCKETLIST

-I INPUT -p udp -m udp --dport 53 -m u32 --u32 0x28=0x2e00 -m comment --comment "Limit RRSIG . queries" -j BUCKETLIST

-A BUCKETLIST -m hashlimit --hashlimit-upto 5/sec --hashlimit-mode srcip --hashlimit-name DNS -m comment --comment "Max rate of this type of request" -j ACCEPT

-A BUCKETLIST -m comment --comment "Drop amplification attacks" -j DROP

 

-------------------------

so now we drop these queries if we see more than 5/second/srcip of these. We here split the traffic into a separate chain (BUCKETLIST), because we don't entries for all our users in the "hashlimit" table -- being afraid there might be limits to how many entries this table can hold. 

This has reduced the load on our DNS-servers enormously. Immediately brought down the qps from peaks of 50K to less than 10K per DNS-server. And we feel confident that we're not blocking any legitimate traffic. Next we want to be more strict, and block the ip-addresses for all queries if they're part of this DDoS. This should be doable using the above rules, plus "ipset". 

When applying firewall rules to a system which you don't have OOB management to, the at command can be used to schedule an allow command incase you lock yourself out.

For example: echo "service iptables stop" | at now + 5 minutes

Network engineers will be familiar with the good old Cisco "reload in 5", used in case you lock yourself out of a router.

Thanks for sharing the tip Jan-Frode.  That's a pretty impressive win in reducing the load on your servers!

Cool indeed!

Small follow-up.. The plan for further blocking abusers is to create two "ipsets":

 

$ cat /etc/sysconfig/ipset

create MANUALBLOCK hash:ip family inet hashsize 1024 maxelem 65536 timeout 604800

create AUTOBLOCK hash:ip family inet hashsize 1024 maxelem 65536 timeout 3600

 

* MANUALBLOCK is a manually maintained list. We add addresses to this list if we see something special we want to block, and these gets blocked for 7 days. 

* AUTOBLOCK is a list maintained by iptables. If a host is detected as doing bad stuff to our DNS-server, they will automatically be added to this list and blocked for 60 minutes.

 

The iptables-rules for managing this is:

:BUCKETLIST - [0:0]

-A INPUT -p tcp -m tcp --dport 53 -m set --match-set MANUALBLOCK src -j DROP

-A INPUT -p udp -m udp --dport 53 -m set --match-set MANUALBLOCK src -j DROP

-A INPUT -p tcp -m tcp --dport 53 -m set --match-set AUTOBLOCK src -j DROP

-A INPUT -p udp -m udp --dport 53 -m set --match-set AUTOBLOCK src -j DROP

-A INPUT -p udp -m udp --dport 53 -m u32 --u32 0x28=0x2e00 -m comment --comment "Limit RRSIG . queries" -j BUCKETLIST

-A INPUT -p udp -m udp --dport 53 -m u32 --u32 0x28=0xff00 -m comment --comment "Limit ANY . queries" -j BUCKETLIST

-A BUCKETLIST -m hashlimit --hashlimit-upto 5/sec --hashlimit-mode srcip --hashlimit-name DNS -m comment --comment "Max rate of this type of request" -j ACCEPT

-A BUCKETLIST -j LOG --log-prefix "IPSET-autoblock: " --log-level 6

-A BUCKETLIST -m comment --comment "Block amplification attackers" -j SET --add-set AUTOBLOCK src

When a customer is added to the AUTOBLOCK list, we log this to syslog, and our NOC sees this so that they can initiate abuse cases against these customers to help them fix their viruses/compromised devices.

Currently only massive amounts of "ANY <ROOT>" and "RRSIG <ROOT>" lookups gets blocked automatically, but I expect we will add more attack signatures here later. F.ex. I want to add a per IP rate-limit, but am a bit worried with how to keep the size of the hashlimit table to a minimum, or if it will handle 100s of thousands of entries.. Will need to investigate more.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.