TomatoUSB

Anyways, I have a lot of other posts describing the problem and potential solutions. I've built a few copies of the firmware using toastman 1.28.7477 for the "mini" target. It works fine on my WRT54GL.

Here's where you can get it.
http://ordorica.org/blog/tomato-firmware-wreduced-bufferbloat

I'm still working to tweak the firmware and figure out what's best. I'd like to have a way to configure the pfifo limit in the GUI instead of hard coding it. SFQ though must be hard coded at compile time.

Can I persuade you make a straight build of 7477.1 K26 MIPSR2 with your mod and post it along with the others?

Thanks!

Sure, I can build it (standard make options), but I won't be able to test it since I don't have the hardware.

I'll get around to it tomorrow. 2AM here.

Be interesting to see how works, just remember the smaller the buffer, the potentially less effective QOS will be less buffer space to reorder (due to priority/bandwidth) limits, my hunch is, this will make the QOS significantly less effective (almost as much as not having any QOS on). I'm guess there probably will need to be a user discovered sweet spot for tuning the best buffer size to meet their latency vs utilisation vs QOS requirements.

Another thing to note is most advice for setting QOS is to set max QOS usable bandwidth to be less than actual bandwidth to prevent buffers ever getting full (and as a result should not effect latency too much), so would also be interesting to see if having smaller buffers would allow higher % of bandwidth to be used.

Cheers, Nick

I've updated the page with a MIPSR2 build.

@N1ckR

The QoS doesn't really reorder, at least not pfifo or sfq.

pfifo does first in, first out. Unless it's a specially marked packet like ECN, or no_nodelay. Most TCP traffic though is unmarked. So it defaults to FIFO.

sfq also pretty much does FIFO, except it also balances bandwidth between each connection (to make sure packets at the bottom of the buffer don't starve for bandwidth).

QoS with the default builds of Tomato (128 packets and 256 packets) is pretty much only QoS using bandwidth limits like you say. Because the buffers are so large, no packets ever get dropped.

IMHO, packets should be dropped so the TCP protocol knows to back-off on speed and keep the link open.

What I've done is more like QoS properly tuned. Instead of just making QoS ineffective, it should be more effective for latency purposes now.

Another neat property of smaller buffers is that even if QoS is set up incorrectly, it now won't lag so much.

Essentially, we could tune for a "default" QoS behavior by specifying packet limits. 256 packets is just too much, unless you have 100Mbit upload.

So is the 10 packet build LESS extreme of a change from the orig or MORE extreme of a change…im assuming the 2 packet is the MOST extreme change from 256 packets?

Also can the following parameters be changed from the webgui or from telnetting into the router?

Changes to the kernel:

pfifo 256 packets -> 2 packets
sfq   128 packets -> 2 packets
enable tcp_ecn
enable tcp_low_latency

Hmm…maybe i should wait and see if toastman gets interested and tests this and maybe decides to include it into his builds for us Mega firmware users

10 packets is a LESS extreme change. You can't change the packet limits from the GUI.

sfq has hard-coded limits set at compile time

pfifo is hard-coded in some script code (but one day it could be changed on the fly with some additional GUI options if people care).

2 packets is optimized for 1Mbit/s uploads and less (30.5ms lag at 768Kb/s)

I chose 768Kb/s because that seemed like a typical North America broadband upload speed. Ideally, you would tweak the buffer limit to the amount of latency you are willing to tolerate.

(Packet Limit * 1500bytes ) / Upload Speed = amount of latency

2 packets gets you around 30.5ms of extra latency for buffering (if your buffers get full).

My upload speed is also around 768 (1Mbps nominal) which is useful. I have put this online with around 20 clients, it's morning now and people are at work, so tonight will be a better test, could be up to 140 users online.

Can you confirm the settings used in the RT-N16 version?

Can you confirm the packet limit changes made it into the build?

log into the router and do: tc -s qdisc

Wow 140 users, that's quite a lot for a little router.

I used this to build it (in src-rt branch):

make r2m V1=7477.1.01_ V2=-reduce-bufferbloat

I'm also curious if pfifo or sfq works better for you, with that many users, and with a low packet limit.

Wow, I just found even more buffers, I think…

linux/net/drivers/net/wireless/bcm43xx/bcm43xx_dma.h

/* DMA engine tuning knobs */
#define BCM43xx_TXRING_SLOTS            512
#define BCM43xx_RXRING_SLOTS            64

/* Suspend the tx queue, if less than this percent slots are free. */
#define BCM43xx_TXSUSPEND_PERCENT       20
/* Resume the tx queue, if more than this percent slots are free. */
#define BCM43xx_TXRESUME_PERCENT        50

If these are the hardware transmit and receive queues, they are quite big by default. If I had ethtool I could confirm, but I don't.

Is there a build of ethool that works on MIPSR1?

If you look, the buffers automatically enable if 50% or less of the tx_queue is full.

That's a huge number of buffers. Before I change it though, I want to confirm with ethool if possible.

I'm guessing the ethernet driver also has large buffers.