I have a lot of clients (140+) on several sites, around 30% of them running BT. We are on ADSL and out upload bandwidth is limited compared with those on cable, so we have to use it wisely.
A couple of pointers - with BT, many connections are not closed by the apps and are left hanging. These are "unexpired" connections. Setting the conntrack expiry times short (really short!) will cut them down considerably. Be careful if you run VOIP as short timeouts can affect the connection. Next, setting your max connections quite low is often better than some high figure. I suggest 2000. Next, we find that most connections from a default "uTorrent" app are UDP, and here at any rate do not produce much in the way of downloads, but consume most of the bandwidth. Same with the newer uTP, which we find to be a real nightmare. Turn off DHT and uTP (in the advanced config) and see if it improves your downloads after a day or two. We all find we get much better download speeds (three times higher actually) by using only TCP connections, which frees up the router. Lastly, long after your torrent application has done with them, many remote sites will still try to connect to your router. These will show up in the "unclassified" group and in the total connection count. Some of these may persist for days, and you can't stop 'em ! Just ignore them.
You can limit the numbers of connections per client in your router's firewall script box. Here are some examples. Don't put these scripts in the FORWARD chain, leave them in PREROUTING, or the router will become unstable when under heavy load from a connection storm.
#Limit no. of TCP connections per user
iptables -t nat -I PREROUTING -p tcp —syn -m iprange —src-range 192.168.1.50-192.168.1.250 -m connlimit —connlimit-above 100 -j DROP
#Limit number of non-TCP connecttions per user
iptables -t nat -I PREROUTING -p ! tcp -m iprange —src-range 192.168.1.50-192.168.1.250 -m connlimit —connlimit-above 50 -j DROP
#Limit number of SMTP connections - useful to stop mass mailer attacks
iptables -t nat -I PREROUTING -p tcp —dport 25 -m connlimit —connlimit-above 5 -j DROP