Contents |
DoS Attacks
Any server can be the recipient of a DoS (Denial of Service) attack. If you have any services running (FTP, HTTP, etc.) these can be attacked. The idea is to tie up the server by making it respond endlessly, or cause buffer overflows, or through injection, etc. to cause the server or its services to become unavailable. Thus, service is denied.
Unfortunately, DoS and distributed DoS (DDoS) attacks are incredibly simple to execute. A simple ping, if run from enough attacking computers, can bring down a server easily.
One of the more common types of attacks centers around making a single attacking system open hundreds or thousands of TCP connections. Another method is to open many web pages continuously. Now, multiply this by several attacking computers (a DDoS) and your sites are not coming up.
Moving to another IP / Provider
Often people mistakenly think moving their server to a new IP range, or even to another hosting provider will stop the DoS attack. This is often just not the case at all.
Moving these sites to another IP address / provider will likely not help you as http attacks are directed against URLs, so wherever they are hosted they will be attacked. You will have to take the same measures regardless of where they are hosted. This is a problem with your sites being attacked, not with the network the sites are on. This is why providers will not simply give out new IPs when servers are being attacked.
Am I being DoSed?
Here are some simple ways to tell:
The first obvious sign is your sites are either not coming up at all, or are extremely slow loading.
Let's check it out:
First, log into the Sago customer portal site at portal.sagonet.com You can list all your devices, and pull up information of each one including a bandwidth graph. Adjust the time frame to be for a day or hour view.
Some attacks will cause an inbound flood of traffic. Others, such as requesting gargantuan quantities of web pages, will show high outbound traffic. If you see a significant spike in either direction far out of proportion to your normal traffic that is a good indication you are being DoSed.
Next, ssh into your server. If you are completely unfamiliar working in the shell, this is a great time to learn.
There are many shell commands to run to collect network status and connection information. Here are merely a few to start with:
[root@host ~]# netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq
-c|sort -nk 1
This will show a summarization of connections per IP. If you see an IP with more than 20 connections, be suspicious.
Also you can list all network connections with something like:
[root@host ~]#netstat -natlp
You can see what ports are being connected to, the number and types of connections, etc.
Google and you will find many tutorials on using netstat and tcpdump to analyze network traffic.
Another great tool is the iptraf program, you can install it with yum install iptraf and analyze the results.
How to Mitigate DoS Attacks
Notice the word mitigate, meaning lessen. Not stop. You are unlikely to be able to stop the attack. It normally ends when the attacker stops it.
What you can do is take steps to lessen the effect and impact on your server. If you lessen the impact enough, your services (web pages) should be up, though maybe a bit slower than normal.
There are many types of attakcs, and this is not a detailed analysis of how attacks work on the network level, but rather presents some tools to stop the most common attack types.
Mitigation Tools
APF:
If you don't already have APF (Advanced Policy Firewall) installed that is a great place to start. See the KB guide on Advanced_Policy_Firewall.
APF contains a feature called Reactive Address Blocking (RAB) to help stop attacks from IPs which demonstrate malicious activities. This is configured in /etc/apf/conf.apf or where ever your APF config is at. There is a lot you can do here, and this is a powerful tool to start with. Read up on it and you just started fighting back.
Let's say you see from the netstat commands above a few IPs with hundreds of connections. You can drop them by:
[root@host ~]# apf -d 85.186.92.4 iptables: No chain/target/match by that name iptables: No chain/target/match by that name Inserted into firewall: Deny all to/from 85.186.92.4
If you later want to remove the block on that IP:
[root@host ~]# apf -u 85.186.92.4 Removed 85.186.92.4 from trust system.
This is only the tip of the APF iceberg - just to get you started. See the APF documentation for full information.
But this simple step alone can stop a lot of attacks with nothing more.
To automate the monitoring and blocking of abusive connections, see the next step.
DoS Deflate Script New!
http://deflate.medialayer.com/
If you have APF installed as above, there is a great script that will run a check every minute and calculate the number of connections per IP. Any IP with greater than 150 connections by default will be added to to the block list in APF.
This is extremely easy to install. Just install APF and then install the DoS Deflate script with:
wget http://www.inetbase.com/scripts/ddos/install.sh chmod 0700 install.sh ./install.sh
Note: This will install a cron job to run every minute which will run a script to analyze and block connections. The tool is installed in
[root@tardis ~]# cd /usr/local/ddos/ [root@tardis ddos]# ls LICENSE ddos.conf ddos.sh ignore.ip.list
And you can see there is a ddos.conf file here - this is where you can adjust the maximum number of connections allowed per IP, length to ban them for, email notifications, etc. We suggest lower it to 75 connections per IP and increasing the block time to :30 (or 1800 seconds)
You can white list IPs by adding them to ignore.ip.list
Uninstalling:
Remove by running the uninstall script:
wget http://www.inetbase.com/scripts/ddos/uninstall.ddos
Testing:
Note: Do not test unless you have access to your server from another IP, or you will just lock yourself out until the timeout expires (default is :10 min)
You can simulate a DoS attack on your server by using the Apache Benchmark tool. This will let you open hundreds or thousands of connections to your server, thus triggering the anti-dos script to block you.
Run the test by running ab from another server with your server to DoS at the target:
archimedes@Anduril:~$ ab -n 500 -c 5 SomeDomain.com
This will request 500 individual pages. Run one or more times.
Now get back on your server and run this command to see all the open connections you just created:
[root@tardis ddos]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
1 Address
1 servers)
30 72.184.142.161
1001 66.118.128.10
You should see a ton of connections coming from your IP. Congratulations! You just DoSed yourself.
Now wait and within 1 minute you will loose your connection to your server when the script runs and blocks you out. Wait :10 or connect from another IP.
Removing a blocked IP using APF:
[root@tardis ~]# apf -u 66.140.128.20
apf(26902): {trust} removed deny all to/from 66.140.128.20
You can view all the IPs being blocked with APF using the -l to list all rules currently in force. apf -? will give other options.
Becoming fluent with APF and DoS Deflate can greatly assist you in combating some forms of DoS attacks. It is well worth the time to learn how to use these tools.
Apache Tools:
There are several Apache modules you can use to help mitigate DoS attacks. Here are some to look into:
1) Mod evasive and mod security are both excellent Apache tools to use. (To clarify, mod secure does not itself really help against DDoS attacks, that is not what it does. Mod evasive can however help. Mode Secure can be used together with other tools however.)
If you have Cpanel these modules are easily installed through WHM.
3) Use other existing Apache tools and modules to help mitigate abusive traffic. For example http://www.apachesecurity.net/tools/ has a CVS containing many useful modules for Apache Security, including httpd-guardian with can help with DDoS and abusive connection traffic.
4) Yet another possible tool to try is: http://sourceforge.net/projects/modbaduser/
The Simple Solution
Another track you can take on all this is to simply identify the site(s) which are the target of the attack and simply cancel them. If it is only one or two sites, then removing them and any DNS reference to your IP could likely resolve the entire situation. Why have all your customer suffer because of one site being attacked? Of course, identify first the correct site and ensure it is an http attack against that SITE and not your IP, etc. first.
But if you find that a certain site is always being attacked - perhaps it is time to exile them from the village.
Advanced Tools
The above tools are what we most commonly use to combat DoS attacks. These are all freely available tools you can use to fight against attacks. These are typically very effective. It takes some time to learn them, but once done you can easily apply them to all your servers.
There are also firewall appliances which you can look at purchasing. These are usually several thousand dollars though and require skill in their configuration and use. We suggest using the free tools covered above and becoming well skilled with them as a good place to start.
We do offer advanced network support, billed at $60 an hour, where we can analyze the traffic and apply the above tools. However, there is no guarantee we can stop every attack - they require individual and customized responses and can be quite time consuming to combat.
Conclusion
There are a number of different methods to identify and block abusive traffic. The tools listed here will go a long way to putting you back on top if implemented fully.
As your provider we want you to have excellent service, and of course want your sites to be up. However, dealing with attacks is the domain of the server administrator, and unless we manage your server this falls outside the bounds of what we generally can do for you.
If you are interested in obtaining a management package, your account manager can assist you with the levels of management available.
Please also understand that there is no simple one-click fix that will resolve abusive traffic. It takes detailed analysis of the situation and then implementing the right tools, monitoring the results and adjusting as needed.
