Installing and Configuring Necessary Software

Bandwidth Limiting HOWTO
Prev |  | Next

# 3. Installing and Configuring Necessary Software

Here, I will explain how to install the necessary software so that we can limit and test the bandwidth usage.

## 3.1. Installing Squid with the delay pools feature

As I mentioned before, Squid has a feature called delay pools, which allows us to control download bandwidth. Unfortunately, in most distributions, Squid is shipped without that feature.

So if you have Squid already installed, I must disappoint you -- you need to uninstall it and do it once again with delay pools enabled in the way I explain below.

1. To get maximum performance from our Squid proxy, it's best to create a separate partition for its cache, called /cache/. Its size should be about 300 megabytes, depending on our needs.

   If you don't know how to make a separate partition, you can create the /cache/ directory on a main partition, but Squid performance can suffer a bit.

2. We add a safe 'squid' user:

   `# useradd -d /cache/ -r -s /dev/null squid >/dev/null 2>&1`

   No one can log in as squid, including root.

3. We download Squid sources from http://www.squid-cache.org

   When I was writing this HOWTO, the latest version was Squid 2.4 stable 1:

   http://www.squid-cache.org/Versions/v2/2.4/squid-2.4.STABLE1-src.tar.gz

4. We unpack everything to `/var/tmp`:

5. `# tar xzpf squid-2.4.STABLE1-src.tar.gz`

6. We compile and install Squid (everthing is in one line):

   `# ./configure --prefix=/opt/squid --exec-prefix=/opt/squid --enable-delay-pools --enable-cache-digests --enable-poll --disable-ident-lookups --enable-truncate --enable-removal-policies`

   `# make all`

   `# make install`

## 3.2. Configuring Squid to use the delay pools feature

1. Configure our squid.conf file (located under /opt/squid/etc/squid.conf):

       #squid.conf
       #Every option in this file is very well documented in the original squid.conf file
       #and on http://www.visolve.com/squidman/Configuration%20Guide.html

       #
       #The ports our Squid will listen on.
       http_port 8080
       icp_port 3130
       #cgi-bins will not be cached.
       acl QUERY urlpath_regex cgi-bin \?
       no_cache deny QUERY
       #Memory the Squid will use. Well, Squid will use far more than that.
       cache_mem 16 MB
       #250 means that Squid will use 250 megabytes of disk space.
       cache_dir ufs /cache 250 16 256

       #Places where Squid's logs will go to.
       cache_log /var/log/squid/cache.log
       cache_access_log /var/log/squid/access.log
       cache_store_log /var/log/squid/store.log
       cache_swap_log /var/log/squid/swap.log
       #How many times to rotate the logs before deleting them.
       #See the FAQ for more info.
       logfile_rotate 10

       redirect_rewrites_host_header off
       cache_replacement_policy GDSF
       acl localnet src 192.168.1.0/255.255.255.0
       acl localhost src 127.0.0.1/255.255.255.255
       acl Safe_ports port 80 443 210 119 70 20 21 1025-65535
       acl CONNECT method CONNECT
       acl all src 0.0.0.0/0.0.0.0
       http_access allow localnet
       http_access allow localhost
       http_access deny !Safe_ports
       http_access deny CONNECT
       http_access deny all
       maximum_object_size 3000 KB
       store_avg_object_size 50 KB

       #Set these if you want your proxy to work in a transparent way.
       #Transparent proxy means you generally don't have to configure all
       #your client's browsers, but hase some drawbacks too.
       #Leaving these uncommented won't do any harm.
       httpd_accel_host virtual
       httpd_accel_port 80
       httpd_accel_with_proxy on
       httpd_accel_uses_host_header on

       #all our LAN users will be seen by external web servers
       #as if they all used Mozilla on Linux. :)
       anonymize_headers deny User-Agent
       fake_user_agent Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6+) Gecko/20011122

       #To make our connection even faster, we put two lines similar
       #to the ones below. They will point a parent proxy server our own Squid
       #will use. Don't forget to change the server to the one that will
       #be fastest for you!
       #Measure pings, traceroutes and so on.
       #Make sure that http and icp ports are correct.

       #Uncomment lines beginning with "cache_peer" if necessary.
       #This is the proxy you are going to use for all connections...
       #cache_peer w3cache.icm.edu.pl parent 8080 3130 no-digest default

       #...except for the connections to addresses and IPs beginning with "!".
       #It's a good idea not to use a higher
       #cache_peer_domain w3cache.icm.edu.pl !.pl !7thguard.net !192.168.1.1

       #This is useful when we want to use the Cache Manager.
       #Copy cachemgr.cgi to cgi-bin of your www server.
       #You can reach it then via a web browser typing
       #the address http://your-web-server/cgi-bin/cachemgr.cgi
       cache_mgr your@email
       cachemgr_passwd secret_password all

       #This is a name of a user our Squid will work as.
       cache_effective_user squid
       cache_effective_group squid

       log_icp_queries off
       buffered_logs on

       #####DELAY POOLS
       #This is the most important part for shaping incoming traffic with Squid
       #For detailed description see squid.conf file or docs at http://www.squid-cache.org

       #We don't want to limit downloads on our local network.
       acl magic_words1 url_regex -i 192.168

       #We want to limit downloads of these type of files
       #Put this all in one line
       acl magic_words2 url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt
       .ram .rm .iso .raw .wav .mov
       #We don't block .html, .gif, .jpg and similar files, because they
       #generally don't consume much bandwidth

       #We want to limit bandwidth during the day, and allow
       #full bandwidth during the night
       #Caution! with the acl below your downloads are likely to break
       #at 23:59. Read the FAQ in this bandwidth if you want to avoid it.
       acl day time 09:00-23:59

       #We have two different delay_pools
       #View Squid documentation to get familiar
       #with delay_pools and delay_class.
       delay_pools 2

       #First delay pool
       #We don't want to delay our local traffic.
       #There are three pool classes; here we will deal only with the second.
       #First delay class (1) of second type (2).
       delay_class 1 2

       #-1/-1 mean that there are no limits.
       delay_parameters 1 -1/-1 -1/-1

       #magic_words1: 192.168 we have set before
       delay_access 1 allow magic_words1

       #Second delay pool.
       #we want to delay downloading files mentioned in magic_words2.
       #Second delay class (2) of second type (2).
       delay_class 2 2

       #The numbers here are values in bytes;
       #we must remember that Squid doesn't consider start/stop bits
       #5000/150000 are values for the whole network
       #5000/120000 are values for the single IP
       #after downloaded files exceed about 150000 bytes,
       #(or even twice or three times as much)
       #they will continue to download at about 5000 bytes/s

       delay_parameters 2 5000/150000 5000/120000
       #We have set day to 09:00-23:59 before.
       delay_access 2 allow day
       delay_access 2 deny !day
       delay_access 2 allow magic_words2

       #EOF

   OK, when we have configured everything, we must make sure everything under `/opt/squid` and `/cache` directories belongs to user 'squid'.

   # mkdir /var/log/squid/

   # chown squid:squid /var/log/squid/

   # chmod 770 /var/log/squid/

   # chown -R squid:squid /opt/squid/

   # chown -R squid:squid /cache/

   Now everything is ready to run Squid. When we do it for the first time, we have to create its cache directories:

   # /opt/squid/bin/squid -z

   We run Squid and check if everything is working. A good tool to do that is IPTraf; you can find it on http://freshmeat.net. Make sure you have set the appropriate proxy in your web browsers (192.168.1.1, port 8080 in our example):

   # /opt/squid/bin/squid

   If everything is working, we add `/opt/squid/bin/squid` line to the end of our initializing scripts. Usually, it can be `/etc/rc.d/rc.local`.

   Other helpful options in Squid may be:

   # /opt/squid/bin/squid -k reconfigure (it reconfigures Squid if we made any changes in its squid.conf file)

   # /opt/squid/bin/squid -help :) self-explanatory

   You can also copy `cachemgr.cgi` to the cgi-bin directory of your WWW server, to make use of a useful Cache Manager.

## 3.3. Solving remaining problems

OK, we have installed Squid and configured it to use delay pools. I bet nobody wants to be restricted, especially our clever LAN users. They will likely try to avoid our limitations, just to download their favourite mp3s a little faster (and thus causing your headache).

I assume that you use IP-masquerade on your LAN so that your users could use IRC, ICQ, e-mail, etc. That's OK, but we must make sure that our LAN users will use our delay pooled Squid to access web pages and use `ftp`.

We can solve most of these problems by using `ipchains` (Linux 2.2.x kernels) or `iptables` (Linux 2.4.x kernels).

### 3.3.1. Linux 2.2.x kernels (ipchains)

We must make sure that nobody will try to cheat and use a proxy server other than ours. Public proxies usually run on 3128 and 8080 ports:

/sbin/ipchains -A input -s 192.168.1.1/24 -d ! 192.168.1.1 3128 -p TCP -j REJECT

/sbin/ipchains -A input -s 192.168.1.1/24 -d ! 192.168.1.1 8080 -p TCP -j REJECT

We must also make sure that nobody will try to cheat and connect to the internet directly (IP-masquerade) to download web pages:

/sbin/ipchains -A input -s 192.168.1.1/24 -d ! 192.168.1.1 80 -p TCP -j REDIRECT 8080

If everything is working, we add these lines to the end of our initializing scripts. Usually, it can be `/etc/rc.d/rc.local`.

We might think to block `ftp` traffic (ports 20 and 21) to force our LAN users to use Squid, but it's not a good idea for at least two reasons:

- Squid is a http proxy with `ftp` support, not a real `ftp` proxy. It can download from `ftp`, it can also upload to some `ftp`, but it can't delete/change name of files on remote `ftp` servers.

  When we block ports 20 and 21, we won't be able to delete/change name of files on remote `ftp` servers.

- IE5.5 has a bug -- it doesn't use a proxy to retrieve the `ftp` directory. Instead it connects directly via IP-masquerade.

  When we block ports 20 and 21, we won't be able to browse through `ftp` directories, using IE5.5.

So, we will block excessive `ftp` downloads using other methods. We will deal with it in chapter 4.

### 3.3.2. Linux 2.4.x kernels (iptables)

We must make sure that nobody will try to cheat and use a proxy server other than ours. Public proxies usually run on 3128 and 8080 ports:

/sbin/iptables -A FORWARD -s 192.168.1.1/24 -d ! 192.168.1.1 --dport 3128 -p TCP -j DROP

/sbin/iptables -A FORWARD -s 192.168.1.1/24 -d ! 192.168.1.1 --dport 8080 -p TCP -j DROP

We must also make sure that nobody will try to cheat and connect to the internet directly (IP-masquerade) to download web pages:

/sbin/iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

If everything is working, we add these lines to the end of our initializing scripts. Usually, it can be `/etc/rc.d/rc.local`.

We might think to block `ftp` traffic (ports 20 and 21) to force our LAN users to use Squid, but it's not a good idea for at least two reasons:

- Squid is a http proxy with `ftp` support, not a real `ftp` proxy. It can download from `ftp`, it can also upload to some `ftp`, but it can't delete/change name of files on remote `ftp` servers.

  When we block ports 20 and 21, we won't be able to delete/change name of files on remote `ftp` servers.

- IE5.5 has a bug -- it doesn't use a proxy to retrieve the `ftp` directory. Instead it connects directly via IP-masquerade.

  When we block ports 20 and 21, our LAN users won't be able to browse through `ftp` directories, using IE5.5.

So, we will block excessive `ftp` downloads using other methods. We will deal with it in chapter 4.

Prev | Home | Next
Before We Start |  | Dealing with Other Bandwidth-consuming Protocols Using CBQ