About the same time as students came back to campus this year, I noticed that my internet connection speed was extremely slow on my gentoo linux notebook. Some websites wouldn't load, and ftp and http downloads never exceded 5KB/s and often were in a measurement not often seen anymore, bits/s. I thought, I know this is a heavy usage period on campus, but this is rediculous.
Well, after our network administrator reported that utilization was was not maxed on our campus partial ds3, I thought perhaps it was a router issue. I started trying different locations, other hosts on campus did not have this problem. I switched to wired, same problem. When others in the building also running linux, were not having the problem I began to suspect by box. But I get full speed at home? Whats the problem?
I did a dslreports speed test which came out rather bizarre, 3434 kbps up and 36 kbps down. :-? Thats two T1's upload speed and a pre 56K modem download speed folks. To which dslreports stated "Your upload speed is much faster than down.. have you tweaked?"
Since the problem was occuring regardless of interface, I began to suspect my kernel. I rebooted with an older 2.6.7 and whamo, the same file that was downloading at 5K/s completed at 200K/s. After the latest gentoo development-sources linux-2.6.8.1, didn't solve the problem I decided to google and found the answer.
The recent 2.6.8 kernels have enabled TCP Window Scaling by default. Window Scaling has been a technique used by cat burglars and the IETF since 1992, see RFC 1323. Basically, it allows for the dynamic setting of tcp window sizes beyond their early fixed limit of 64K to increase performance on the Internet with modern equipment. So why doesn't it work with Linux? Well the problem is not with Linux at all, other than the fact that they turned it on by default. Apparently many routers and packet firewalls are rewriting the window scaling factor during a transmission, instead of only during the initial handshake (SYN). This means that the sending and receiving side are assuming a different TCP window size. The result of this misnegotiation of protocol, is very slow successful traffic if at all.
This also explains why the problem is visible on some sending and receiving sites, because only devices behind the path of broken routers are affected. For instance, why my notebook worked fine from my house, or why I was able to get to some sites from on campus at full speed. Also apparently some routers are only mangling in one direction, which would explain that crazy speed test above.
The solution? Well, some of the linux developers are hoping that leaving the option enabled will force the issue, so that vendors will fix their routers. As for me, I was able to follow David S. Miller's suggestion to turn off the feature dynamically in the kernel.
The following command will disable the win scaling feature for the running kernel:
sysctl -w net.ipv4.tcp_default_win_scale=0
And the following command will make sure it gets set next reboot:
echo "net.ipv4.tcp_default_win_scale=0" >> /etc/sysctl.conf
In case you hadn't picked up on it, this is not a gentoo specific issue. Redhat fedora users, you might be affected as well, along with any other distribution using the recent stock 2.6.8 kernels. This LWN article is the only press I have seen about the problem. For a more complete discussion of the topic, here's the start of the thread on the Linux Kernel Mailing List. It would be nice if someone had a complete list of affected routers, some have mentioned openbsd and cisco.
