In a constant run to optimize resources all of us tend to find a way to fine-tune different aspects of the systems to achieve better server performance. While some people are getting satisfied with minor adjustments in the applications’ configuration, more advanced guys try to go as deep as possible and alter low level kernel flags to try and get the best of the best. That’s all kinda cool and fun, but as it is impossible know everything about everything we often ask google for advice, and that’s where some problem begin. There are bunch of howto’s out there on the internet with different solutions on improving performance, but most of them don’t go deep enough to show possible drawbacks of such solutions. As sysadmins a lazy people, RTFM or better to say RTF-RFC and getting all the internals is not in our habit, until something brakes. And that’s what I had to face yesterday.
My brother called me for help with his problem that he was trying to figure out for some time already, but it seemed that nothing could save him. The problem was pretty tricky and it was pretty well described in Leonid’s blog post, so I will get more on how it was rectified:
The troubleshooting started with simple things and went all the way down to tcpdump. Given the fact that his office server could successfully communicate with Amazon server, but non of the desktops/laptops behind the office server couldn’t, the fun begin.
– traceroute shows that we have proper connectivity and routing in place.
– tcptraceroute shows problems, netcat confirms the problems with connection timeout
– iptables rules look fine on all the parties
– logs on the Amazon server do no show anything useful
My first thought was: WTF! And when we have WTF related to connectivity, we use tcpdump.
The tcpdump shows that:
– initial TCP SYN packet successfully leaves office laptop
– the same packet successfully passes the firewall coming to LAN interface and leaving the WAN interface
– and the TCP SYN packet successfully arrives to the destination Amazon server, but
– there is no TCP SYN-ACK reply ever leaving Amazon server towards office
In case when we leave office laptop alone and try to do the same thing from the office server, both TCP SYN successfully reaching the Amazon server and we have TCP SYN-ACK as well as any following TCP packets successfully traveling between the communicating nodes.
After we have all of the above info gathered, the problem was localized to Amazon server and the question was as simple as: why Amazon server is not replying with TCP SYN-ACK to the office laptop, while it does reply with TCP SYN-ACK to everyone else. That was the point where my knowledge of the TCP internals was exhausted and I turned to google for a solution. As always, there are bunch of articles out there all with different ideas and very limited low-level explanations, so came back to tcpdump on Amazon server and started the game of “find 3 differences between two TCP ACK packets that arrive one from office laptop and one from office server”. The only two differences I managed to see were:
– TCP window size of packet from laptop was way bigger (29200) then from office server (5840)
– Timestamp value of packet from laptop was way smaller (64389040) then from office server (809044567)
Quote from tcpdump, first packet from laptop, second from office server:
xxx.xxx.xxx.xxx.55470 >yyy.yyy.yyy.yyy.22: Flags [S], cksum 0x8cb3 (correct), seq 3904091306, win 29200, options [mss 1460,sackOK,TS val 64393040 ecr 0,nop,wscale 8], length 0
15:53:00.755020 IP (tos 0x0, ttl 50, id 55870, offset 0, flags [DF], proto TCP (6), length 60)
zzz.zzz.zzz.zzz.43952 > yyy.yyy.yyy.yyy.22: Flags [S], cksum 0xcfbf (correct), seq 1790824553, win 5840, options [mss 1460,sackOK,TS val 809044567 ecr 0,nop,wscale 8], length 0
15:53:00.755071 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
With the above two facts I started the investigation on TCP window size. I did remember that this metric can be dynamic and the difference is possible, but I thought it might be more problem then timestamp, that is obviously different all the time and who cares about timestamp anyway. Google showed me a number of options to try with regards to sysctl, including but nor limited to disabling TCP time scaling, adjusting different buffers of OS TCP stuck and so on, which I tried to apply everywhere including Amazon server, office server and office laptop all with no success. Finally, some post via google (lost original post) told that setting net.ipv4.tcp_tw_recycle to 0 solved the problem. Having no other alternatives I did apply the setting on Amazon server and all came back to normal – now everyone could connect to the server and all was working as it supposed to.
Since the problem was gone, I reported to my bother that he can continue with his other tasks as one problem less, made sure that the flag is set permanently in /etc/sysctl.conf and realized that now I need to learn more of TCP internals. Fortunately there is an amazing article by Vincent Bernat “Coping with the TCP TIME-WAIT state on busy Linux servers” that dives into how the whole thing works, why we should not mess with the TCP TIME-WAIT and that at the end, changing this flag will not give one any visible advantages.
As a resume of the above, before you change any kernel flags, make sure you really understand what you are doing. Before applying any configuration changes proposed by some online howto, make sure you exactly what you are doing and don’t trust anyone blindly. Finally – learn to troubleshoot with low-level tools that will help you spot the problem or at least show the directions for further troubleshooting.
P.S.: Leonid, thanks for fun experience and something new! Was fun!