Boosting Socket Performance on Linux

Cop writes "The Sockets API lets you develop client and server applications that can communicate across a local network or across the world via the Internet. Like any API, you can use the Sockets API in ways that promote high performance -- or inhibit it. This article explores four ways to use the Sockets API to squeeze the greatest performance out your application and to tune the GNU/Linux® environment to achieve the best results."

Be aware

Some engineers at Berkeley have been looking at this for a while, but haven't gotten much credit for it.

Re:Be aware

Exactly... especially with things like these, it's usually best for the entire internet if you just stick with the defaults... they are defaults for a reason, it might not be the best for you, but it's most likely the best for the internet as a whole.

Reminds me of those people tweaking firefox settings to hammer all kind of webservers... sure, your browsing might be a slight bit faster, at the expense of the browsing of lots of other people...

Re:Be aware

Looks like a rip off of an OnLamp [onlamp.com] article from a few months ago, and not a very good one at that! At least the OnLamp [onlamp.com] article explained how to tweak a few more OS's and the math was correct. And just to add insult to injury the article on OnLamp was written by one of those Berkeley guys [lbl.gov] ;-)

slashdotted?

Judging by the response time from IBM's web server, it looks like they have yet to put their advice into practice.

GNU/Linux®...A lessefficent way to say Linux

I mean really, I think we understand what you mean by just saying Linux.

Hello 1995

This reads like an article from the 90's. This being 2006 and all, I would hope that programmers know how to make effective use of TCP/IP sockets. I wonder if maybe they just yanked an article from 1995 and did a search/replace on s/Windows/GNU Linux/g.

Re:Hello 1995

Agreed. In fact, as someone who learned socket coding around 1999/2000 (and as a result do not have a good grasp on how to actively define register variables, compilers do that stuff for you these days) I did all of these things out of habit, and didnt fully understand them until this article.

In the same line - where is the discussion of different FD table polling mechanisms? select() versus poll(), and wheres the writeup about Linux's epoll(). I would have been interested in an epoll() article, especially how it compares to FreeBSD's kqueue().

Re:Hello 1995

In the same line - where is the discussion of different FD table polling mechanisms? select() versus poll(), and wheres the writeup about Linux's epoll(). I would have been interested in an epoll() article, especially how it compares to FreeBSD's kqueue().

For the overview, you want Dan Kegel's c10k page:

http://www.kegel.com/c10k.html [kegel.com]

Hello 2003.

The paper is 2 years, 2 months old. Many of the arguments will still be valid, but the code in all cases will have evolved considerably. In addition, other code has certainly been developed (there's a hard real-time UDP patch for Linux [uni-hannover.de], for example) and the state of affairs is - if anything - much more muddled today.

Documentation like this is great and extremely valuable. It would be much more valuable, however, if it remained current. For example, can the ABISS [sourceforge.net] project (which improves block I/O) be used at all? What do the numbers look like, when using profiling tools like Web100 [web100.org] (which profiles TCP communications)?

Has anyone run the Linux or one of the *BSD kernels through DAKOTA [sandia.gov], KOJAK [fz-juelich.de] or PAPI [utk.edu] to determine where, precisely, bottlenecks are within the kernels? It's easy to theorise, but isn't it cleaner to measure?

Now, I'm not saying these things aren't being done. They probably are, somewhere, by someone, but if the results aren't getting published we don't really know what impact what changes are going to have. The current method of evolving Operating System code in general is often a mix of personal theory and subjective experience based on non-random samples of activity. That can't really be a good way to do things, can it?

If I'm wrong, feel free to say. If I'm right, then maybe it would be a good thing if someone (possibly me) put together some kind of testing kit for measuring Linux kernel performance and actually measured the stats for Linux kernels on some kind of regular basis.

Re:Hello 1995

The Linux kernel automatically doubles the buffer for its own use. In the article:

Within the Linux 2.6 kernel, the window size for the send buffer is taken as defined by the user in the call, but the receive buffer is doubled automatically. You can verify the size of each buffer using the getsockopt call.

From the MAN page [linuxmanpages.com]:

NOTES

Linux assumes that half of the send/receive buffer is used for internal kernel structures; thus the sysctls are twice what can be observed on the wire.

The article could have better explained that in context. For the most part it's automatic though, so don't worry about it.

Summary ripped directly from article (again)

Here is the summary:

The Sockets API lets you develop client and server applications that can communicate across a local network or across the world via the Internet. Like any API, you can use the Sockets API in ways that promote high performance -- or inhibit it. This article explores four ways to use the Sockets API to squeeze the greatest performance out your application and to tune the GNU/Linux® environment to achieve the best results.

Here is the first paragraph of the article:

The Sockets API lets you develop client and server applications that can communicate across a local network or across the world via the Internet. Like any API, you can use the Sockets API in ways that promote high performance -- or inhibit it. This article explores four ways to use the Sockets API to squeeze the greatest performance out your application and to tune the GNU/Linux® environment to achieve the best results.

Unless Cop (the submitter) is actually M. Tim Jones (the article author), Cop didn't write a darn thing.

Didn't we just have this discussion on /. a few days ago?

No mention of alternatives to select?

Tuning socket parameters is great and all, but the real performance problem with socket IO has to do with using select and poll. There are high-performance alternatives (which admittedly tend to vary from OS to OS) that are so far superior that I wouldn't even consider the default methods unless complete code portability were a crucial factor.

Re:Code Portability

There was a Boost library in the works to encapsulate all of this rather nicely, but I'm not sure if it ever made it out of beta. ACE is another option, though that tends to be overkill for some projects. I rolled my own class wrapper around this stuff, but then I enjoy library programming.

Re:No mention of alternatives to select?

Try this:

http://www.xmailserver.org/linux-patches/nio-impro ve.html [xmailserver.org] /dev/epoll

The website is hideous, but there used to be benchmarks against different polling/selecting methods. If I remember correctly, its kinda trial and error, YMMV, kind of stuff. Its worth a look.

Nothing new

going from Socket 7 to Socket 462 to Socket 478 boosted it quite a damn bit over the years.

GNU/Linux®?

Why the corporate-style circle-R? Is this a subtle bit of sarcasm or trollery targeting RMS's followers?

IBM is getting some good Linux content...

...on developerWorks, not the least of which, if I may say so, is the GLib tutorial [ibm.com] I wrote for them this past summer. If you wanted how to use various GLib collections and utilities - lists, tables, trees, quarks, relations, and all that - check it out. You can even download a nice PDF file for offline perusing.

Folks who are thinking about writing something technical - give dW a shot. The editors are savvy folks and there's lots of good stuff up there already.

Oh, and book plug [pmdapplied.com]!

I've always wanted to know if it is possible

to send signals to a network socket without writing code but using some ready made command-line tool (netstat?)? I've looked around for this but can't seem to find anything...

Re:I've always wanted to know if it is possible

which acts very much like cat

It ignores you except at feeding time, and pees in your shoes when it's mad at you?

Re:I've always wanted to know if it is possible

Netcat might be what you want. It has two modes, a "client" and "server" mode. In client mode, it connects to an IP/port that you specify, then reads data from stdin and sends it through that socket. In server mode, it listens on a port you specify, and prints any data it received to stdout.

Is that what you're looking for?

Nagle's algorithm

For an application where I want both low latency AND high bandwidth, it's not enough to leave Nagle's algorithm on or off. If I leave it on, I'll get increased bandwidth, but >200ms latency due to the Nagle delay. If I leave it off, I get low latency, but the computer will (typically?) send out one network packet per send() call, which means inefficient use of bandwidth unless the calling code is very careful to call send() only with large amounts of data per call.

To get around the above problems, I came up with the following scheme: Leave Nagle's algorithm enabled, but create a FlushSocket() function that merely disables Nagle on the socket, then calls send() on the socket with a 0-byte buffer, then enables Nagle again. This apparently forces the TCP stack to immediately send any data that it may have accumulated in its Nagle-buffer. Therefore the only thing the calling code has to remember to do is to call FlushSocket() whenever it has called send() one or more times and doesn't think it will be sending any more data any time soon.

The above technique seems to work pretty well under Linux, Windows, and OS/X (and is more portable than Linux-specific flags like TCP_CORK, etc), but I haven't seen it documented anywhere. Is that simply an oversight, or is there some nasty downside to this technique that I'm overlooking?

flush( sd ) would be nice

Wouldn't it be nice if C programmers were given an option similar to what fflush does for streams? Something like flush(sd) whenever you need to ignore Nagle's algorithm. In this way you can enable and disable nagling dynamically in your program without calling setsockopt to switch nagling on and off. This option is given for Java since you can easily convert a socket to any type of stream you wish, while most Stream objects have a member function flush(). Perhaps I am wrong and such an interface is already provided in C but I personally never found one, while the necessity for it appears to be obvious.

Math error in paper?

throughput = window_size / RTT

110KB / 0.050 = 2.2MBps

If instead you use the window size calculated above, you get a whopping 31.25MBps, as shown here:

625KB / 0.050 = 31.25MBps


That's funny, I get 12.5MBps

???

Socket tuning

I've used Azureus a lot on my Linux box, and one of its features is tunability and graphs. Number of connections, max up and down, etc, and watch the results. Now, I have a very asymmetric line (10:1 ratio). I've noticed that trying to use maximum upload and download at once can create sinewave patterns of slow response that look a lot like resonant feedback, and in extreme cases can wedge the line completely, throughput zero on all net apps. Running uploads at 20K and leaving the top 5K unused gets a far better total rate both up and down.

What I'm wondering is, might it be possible to make these sort of calculations in kernel, detect congestion feedback and back off automatically? I'm not talking about the regular exponential backoff algorithm, but about some sort of best-rate prediction based on detecting the characteristic shape of feedback waves and backing off until they disappear.

GNU/Linux®

GNU/Linux® ®? WTF®

Always liked the Winsock Lame List

Of course it is rather windows centric, but most of the issues apply across platforms (only a few talk about WSA functions)

However Lame List [tangentsoft.net] contains a lot of wonderful nuggets.

I must disagree with the article however, there are so SO few times that disabling the Nagle algorythm is the correct answer that the standard answer when someone asks about it on the networking forums is that the asker doesn't understand Nagle, and to reenable it. Telnet is even a bastard case in that your networking performance may actually go UP sending smaller bursts of network characters, rather than one at a time, each in its own packet. But you have to measure your own performance.

Frankly none of these suggestions will get you ultimate performance from a 10 Gig networking stack, and that is where networking finally becomes fun

The trouble with the Nagle algorithm

I really should fix the bad interaction between the "Nagle algorithm" and "delayed ACKs". Both ideas went into TCP around the same time, and the interaction is terrible. That fixed timer for ACKs is all wrong.

Here's the real problem, and its solution.

The concept behind delayed ACKs is to bet, when receiving some data from the net, that the local application will send a reply very soon. So there's no need to send an ACK immediately; the ACK can be piggybacked on the next data going the other way. If that doesn't happen, after a 500ms delay, an ACK is sent anyway.

The concept behind the Nagle algorithm is that if the sender is doing very tiny writes (like single bytes, from Telnet), there's no reason to have more than one packet outstanding on the connection. This prevents slow links from choking with huge numbers of outstanding tinygrams.

Both are reasonable. But they interact badly in the case where an application does two or more small writes to a socket, then waits for a reply. (X-Windows is notorious for this.) When an application does that, the first write results in an immediate packet send. The second write is held up until the first is acknowledged. But because of the delayed ACK strategy, that acknowledgement is held up for 500ms. This adds 500ms of latency to the transaction, even on a LAN.

The real problem is that 500ms unconditional delay. (Why 500ms? That was a reasonable response time for a time-sharing system of the 1980s.) As mentioned above, delaying an ACK is a bet that the local application will reply to the data just received. Some apps, like character echo in Telnet servers, do respond every time. Others, like X-Windows "clients" (really servers, but X is backwards about this), only reply some of the time.

TCP has no strategy to decide whether it's winning or losing those bets. That's the real problem.

The right answer is that TCP should keep track of whether delayed ACKs are "winning" or "losing". A "win" is when, before the 500ms timer runs out, the application replies. Any needed ACK is then coalesced with the next outgoing data packet. A "lose" is when the 500ms timer runs out and the delayed ACK has to be sent anyway. There should be a counter in TCP, incremented on "wins", and reset to 0 on "loses". Only when the counter exceeds some number (5 or so), should ACKs be delayed. That would eliminate the problem automatically, and the need to turn the "Nagle algorithm" on and off.

So that's the proper fix, at the TCP internals level. But I haven't done TCP internals in years, and really don't want to get back into that. If anyone is working on TCP internals for Linux today, I can be reached at the e-mail address above. This really should be fixed, since it's been annoying people for 20 years and it's not a tough thing to fix.

The user-level solution is to avoid write-write-read sequences on sockets. write-read-write-read is fine. write-write-write is fine. But write-write-read is a killer. So, if you can, buffer up your little writes to TCP and send them all at once. Using the standard UNIX I/O package and flushing write before each read usually works.

John Nagle

...what about UDP?

the article is all about TCP, which is great. how about an article on optimizing UDP though?

Pining for Doors

What really bums me out about doing network services on the Linux platform is that Linux does not support doors, a la Solaris, so you can't have multiple processes collaborating on a single socket service without a scheduler burp. There was a guy who implemented doors for 2.4, but his code was never adopted into the kernel, and now its rotting away....

Linux is quite tragic that way. Hopefully there will be a Debian user-land on the OpenSolaris kernel soon, and then I can rock-n-roll again.

Re:somewhat old...

UDP is lacking a lot of features that TCP has. Such as resend on lost transmission.

I agree though, nothing earth shaking. Nagle's algorithm is discussed in depth in most TCP/IP books, and so is how to turn it off. Wake me up when they post something new.

Re:Never trust an article with a (R) symbol...

Linux IS a registered trademark, you know. Especially if you are an Australian...