Stories
Slash Boxes
Comments

News for nerds, stuff that matters

Benchmarking the Scalability of BSD and Linux

Posted by michael on Sun Oct 19, 2003 03:10 AM
from the pedal-to-the-metal dept.
Fefe writes "I recently did some benchmarks for a talk about scalable network programming I held at Linux Kongress 2003. The benchmark results turned out to be surprising enough to present them on their own. This ought to end those pesky flame wars about whose IP stack or memory management scales better. Or maybe not."
This discussion has been archived. No new comments can be posted.
Display Options Threshold:
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • FreeBSD may be dying but it's fast! (Score:5, Interesting)

    by seanadams.com (463190) * on Sunday October 19 2003, @03:10AM (#7252725)
    (http://www.seanadams.com/)
    Can anyone explain the discontinuities in the FreeBSD plots? Intuitively I would guess that something is breaking at high load, rather than getting miraculously faster. The author suggests that a clever optimization is kicking in, but I wonder if his tests were actually ensuring that the calls succeed.

    Also watch out as you read the graphs - just to keep you on your toes, he changes the colors in every one!

  • no! (Score:2)

    by millette (56354) <.moc.olgaw. .ta. .ettellim.> on Sunday October 19 2003, @03:12AM (#7252734)
    (http://rym.waglo.com/ | Last Journal: Monday May 10 2004, @12:11PM)
    Who's going to publish benchmark about this webserver's scalability?
    • Re:no! by PhlegmMaster (Score:2) Sunday October 19 2003, @04:27AM
    • Re:no! by unixformat (Score:1) Sunday October 19 2003, @05:23AM
    • 2 replies beneath your current threshold.
  • Got Mirror? (Score:2, Funny)

    by Anonymous Coward on Sunday October 19 2003, @03:17AM (#7252745)

    Apparently the webserver with the results didn't scale so well. It's /.ed already.

    • Re:Got Mirror? by Stephen Samuel (Score:2) Sunday October 19 2003, @03:46AM
    • Re:Got Mirror? by TheBadger (Score:1) Sunday October 19 2003, @05:23AM
    • 1 reply beneath your current threshold.
  • Isn't it ironic.... (Score:1, Funny)

    by Anonymous Coward on Sunday October 19 2003, @03:20AM (#7252749)
    That there is a story posted on scalability, but the gents web server does not respond?!? Just a thought...
  • Why the hell fbsd 5.1? (Score:1, Insightful)

    by Anonymous Coward on Sunday October 19 2003, @03:21AM (#7252753)
    This guy either had an agenda, or knows nothing about freebsd. He should have used freebsd 4.8, or 4.9rc2. Why the hell use 5.1-current? It's still experimental and has not gone -STABLE yet.

    His experiments noted a few crashes. HELLOOOOO. That's a clue. Check out this line from the study:
    I installed a FreeBSD 5.1-RELEASE on a free partition. However, the kernel turned out to be unstable under load and would panic or even freeze under load. So I reluctantly upgraded the kernel to 5.1-CURRENT
    Reluctantly? Why reluctant? He's already installed the non-stable branch. Why not go whole hog and run current? It's like he was trying to make freebsd fail by installing an experimental non-stable branch.

    Sheeeez. Just for those who don't know, it's important in freebsd to not just look at the version numbers, but also look at what the branch is -RELEASE, -CURRENT or -STABLE. Right now, only the 4.x branch is -STABLE. Just because the version number is higher does not mean you should upgrade. Stick with -STABLE, unless you know what you're doing

    Now, I'm not saying linux won't do a great job compared to freebsd. It's just that this study did everything it could to slant against freebsd. What kind of reaction would the community have it this had been released by Microsoft? I suspect in this case, it was just a beginner's mistake.

    Benchmarking is hard work. He should redo his study with proper installations from a -STABLE tree, and report back in.
    • Re:Why the hell fbsd 5.1? by Anonymous Coward (Score:1) Sunday October 19 2003, @04:22AM
      • 1 reply beneath your current threshold.
    • I knew when I read the post that this would lead to another FreeBSD v. Linux flamewar, despite the author's claims of 'hoping' to end those.

      I don't think using 5.1-CURRENT is a problem, but the way the benchmark results were layed out was begging for a flamewar. As I explained earlier [slashdot.org] the results are not as bad as either linux fanatics, or FreeBSD fanatics would have it. It would have been simple to avoid such flamewar (or am I too optimistic?) by doing two things:

      Explain the status of both (linux 2.6 and FreeBSD 5.1) development branches - as I have outlined in my earlier post. If you take into consideration what I have written above, than you would have realized that results for FreeBSD are not that bad, in fact, they are excellent.

      Include results for 4.8 - or 4.9 rc3 (but I would be happier with just the production release) in the test, just as 2.4 was included on the linux side.

      To sum up: I believe that these benchmarks confirms what I thought for a long time: FBSD 5.1 development is on par with Linux 2.6. Perhaps this was the reason for his last "Or may be not" remark.
      [ Parent ]
      • 1 reply beneath your current threshold.
    • Re:Why the hell fbsd 5.1? by Zefram (Score:1) Sunday October 19 2003, @07:46AM
    • Re:Why the hell fbsd 5.1? by You're All Wrong (Score:1) Sunday October 19 2003, @08:03AM
    • Re:Why the hell fbsd 5.1? by nutznboltz (Score:2) Sunday October 19 2003, @08:47AM
    • Re:Why the hell fbsd 5.1? by Uerige (Score:1) Sunday October 19 2003, @09:18AM
    • Re:Why the hell fbsd 5.1? by Jungle guy (Score:2) Sunday October 19 2003, @01:01PM
    • 5 replies beneath your current threshold.
  • great (Score:1)

    by andih8u (639841) on Sunday October 19 2003, @03:26AM (#7252770)
    Note to self: Never run benchmarking tests on live webserver

    • 1 reply beneath your current threshold.
  • It looks as though (Score:1)

    by maroberts (15852) on Sunday October 19 2003, @03:26AM (#7252771)
    (http://www.mainecoon.plus.com/ | Last Journal: Wednesday January 17 2007, @11:05AM)
    ...his web server is not very scaleable in response to Slashdotting!
  • The winner in this case is Open Source software.

    The article is very fair and very well thought out. It is almost like reading a research paper. It looks like he is inviting criticism, insight, and corrections, rather than trying to force the experiments into a pre-determined outcome.

    Such a thing is not possible in the proprietary world. Any study done on proprietary software has to be tainted with opinion and the experiments must be skewed. Read the EULAs. Some EULAs won't even allow you to publish the results of such tests.

    Open Source software, of the BSD kind and the GPL kind, has totally changed the way we think about and work with software. One day, we will be able to scientifically determine what software we need to suit our needs. We will know ahead of time exactly what limits and what capabilities each piece of software has. IT managers will be able to sort through real facts based on real research, rather than a bunch of shallow articles and biased reports. Software will survive on its merits alone.

    The whole industry is going to benefit by this, in a large, large way. The question one day will no longer be "Microsoft or Linux?" but "Which Open Source software should we use, and why?"
  • Mirror (Score:1)

    by infonick (679715) on Sunday October 19 2003, @03:28AM (#7252776)
    (http://www.infonick.net/)
    I have a mirror (minus four graphs) :( workin on it...

    members.shaw.ca/infonick [members.shaw.ca]
    • Re:Mirror by infonick (Score:1) Sunday October 19 2003, @03:39AM
    • Re:Mirror by infonick (Score:1) Sunday October 19 2003, @03:43AM
  • by cnb (146606) on Sunday October 19 2003, @03:29AM (#7252781)
    (http://freedomink.org/)
    it appears, fefe's website does not scale well to a slashdotting. :)

    • 1 reply beneath your current threshold.
  • by Krapangor (533950) on Sunday October 19 2003, @03:33AM (#7252789)
    (http://www.slashdot.org/~Krapangor)
    Win: Linux is better than BSD.
    Loose: Linux can't obviously not contain any SCO IP, because it's not business ready. SCO's lawsuit will collapse.

    The only problem is that the BSD benchmarking might count as disturbing the rest of the dead and will have severe legal consequences.

  • Don't miss to notice the recursion... (Score:3, Interesting)

    by presroi (657709) <neubau@presroi.de> on Sunday October 19 2003, @03:35AM (#7252796)
    (http://www.mathiasschindler.de/)
    Topic of this paper [bulk.fefe.de] is

    "Scalable Network Programming
    Or: The Quest For A Good Web Server (That Survives Slashdot)"


    What a coincidence!

    By the way, fnord web server [www.fefe.de] has at least once survived one slashdotting-event. 4 seconds of googleing result in this comment [slashdot.org] which should have let to a stream of visitors.

    I hope fefe will publish the numbers of visitors and the behavior of its web server as soon as possible.

    • 1 reply beneath your current threshold.
  • by gid-goo (52690) on Sunday October 19 2003, @03:38AM (#7252806)
    Who wants to start taking bets on when Theo takes the bait.
  • by Zakabog (603757) <zakabog@noSpAM.elitehunters.com> on Sunday October 19 2003, @03:38AM (#7252808)
    Slashdot's scalability is quite amazing. It seems as a sites ability to resist a slashdotting goes up, slashdot's ability to slashdot a website also goes up. Usually it's at a higher rate but sometimes the sever makes it out alive. Although most of the time it ends up as a smoking pile of slag.
  • I have a complete copy (graphs and all) here [ofdoom.com].

  • Interesting... (Score:1)

    by Ianoo (711633) on Sunday October 19 2003, @03:46AM (#7252830)
    (Last Journal: Friday May 21 2004, @10:08AM)
    Although the results compared with BSD are fairly interesting, the results difference between Linux 2.4 & 2.6 are truely astounding. For those who can't see the graphs or can't be bothered to RTFA...

    bind() to port 0
    n = number of processes:
    2.4: O(n) rise number of processes
    2.6: around 0 even with 10000 ports

    nmap every other page in a 200MB file:
    n = latency in microseconds:
    2.4: O(n) rise
    2.6: constant (basically no latency)

    The kernel guys have certainly done some fantastic work in this last year. I can't wait for 2.6.0!
  • Well (Score:2)

    by arvindn (542080) on Sunday October 19 2003, @03:47AM (#7252833)
    (http://arvindn.livejournal.com/ | Last Journal: Monday June 16 2003, @12:39AM)
    I wasn't able to see the benchmark results, but I was able to see that the server its hosted on is not very scalable.
  • by njdj (458173) on Sunday October 19 2003, @03:54AM (#7252852)
    I hope RMS reads the slides. They're in German at the link I used, so here's a translation of slide 13 which is page 14 of the PDF file:

    "The memory required for an empty process is shockingly large on current Linux systems. However, this is not the fault of Linux, it's the fault of GNU libc.

    GNU libc leads all libc implementations by a large margin in bloat and waste of memory. One day it got so painful that I wrote my own libc. With this, a static binary of 'Hello world' took only 300 bytes..."

    I've long suspected that FSF stands for Fat Software Foundation.

    (He doesn't say, but I assume his home-brew libc was a subset, otherwise we'd all want it).
  • My benchmarks. (Score:3, Funny)

    by Black Parrot (19622) on Sunday October 19 2003, @03:57AM (#7252859)


    I took three computers out in my rowboat, a Windows system, a Linux system, and a BSD system, and threw them overboard to see what would happen.

    The Windows system sank like a rock, the Linux system bobbed back to the surface, and the BSD system rose to the sky, to be greated by a chorus of angels.

    Then I woke up, so I don't know what the angels were singing.

  • Wrong FreeBSD version used (Score:2, Informative)

    by Anonymous Coward on Sunday October 19 2003, @03:58AM (#7252863)
    From the 5.1-RELEASE notes:

    Experimental 1:1 and M:N thread libraries...
    Experimental Name Service Switch infrastructure...
    Experimental support ...

    Although stability is greatly improved and many bugs have been fixed, FreeBSD 5.1 might not be suitable for ...

    So this guy grabs a beta version of a new tree in freebsd, and runs it against stable netbsd, openbsd and linux? Eh? Did he even compile his own kernel and take out all the debugging information in the released kernel? Did he turn off the debug info in the kernel config, on by default? I kinda doubt it somehow... If he knew enough to do that, he'd know that 5.1 has NOT been tagged stable.

    Most freebsd users will look at this study and laugh. But people who don't know anything about freebsd (perhaps that includes the author?) might get the wrong idea. The study needs to be redone using the -stable tree, not the debug-riddled 5.1-RELEASE.
  • kqueue() is available on the production quality 4.x branch of FreeBSD, whereas epoll() is only available on the 2.6 development branch of Linux and also requires a 2.3 glibc compiled specifically with support for epoll(). The glibc currently in Debian unstable, for example, still doesn't support epoll.

    The SIGIO comparison is somewhat fair, but keep in mind that implementing a program with SIGIO requires multiplexing I/O asynchronously, which radically alters program design. Linux 2.4 lacks a synchronous, stateful interface for I/O mulitplexing. Programs that implement synchronous I/O multiplexing can easily transition to a library for their main loop to immediately take advantage of high performance multiplexing/events mechanisms. One such library is libevent [monkey.org].

  • From the article:

    • OpenBSD 3.4 was a real stinker in these tests. The installation routine sucks, the disk performance sucks, the kernel was unstable, and in the network scalability department it was even outperformed by it's father, NetBSD. OpenBSD also gets points deducted for the sabotage they did to their IPv6 stack. If you are using OpenBSD, you should move away now.

    Sabotage? Someone able to bring me up to speed on this?

    As for the installation routine, it is something out of the dark ages, considering even redhat 4.0 - my first Linux installation - had a curses based install. However, if you ignore that, it still seems a decent enough platform - I use it on an Ultra5 at work and as my firewall/NAT gateway at home.

    What I'd now like to see, which is what often doesn't occur, is for each OS camp to provide tuning suggestions to the researcher, and have him re-perform the tests. This is because no production webserver is likely to be untuned from the default installation.

    Oh and the read-only sysctl problem for FreeBSD that he mentions was probably due to securelevel's being on (meaning you can't modify kernel variables).

  • An extended troll against OpenBSD (Score:2, Flamebait)

    by dmiller (581) <djm@mind r o t . o rg> on Sunday October 19 2003, @04:11AM (#7252892)
    (http://www.mindrot.org/)

    Using an unstable development version and then complaing about instability, peppering the results with emotive commentary and clueless rhetoric. (btw the 1024-cylinder boot restriction he complains so much about has been fixed for a while) Especially funny was this idiotic statement:

    OpenBSD also caused a lot of grief on the IPv6 front. The OpenBSD guys intentionally broke their IPv6 stack to not allow IPv4 connections to and from IPv6 sockets using the IPv4 mapped addresses that the IPv6 standard defines for thus purpose. I find this behaviour of pissing on internet standards despicable and unworthy of free operating systems.
    Someone should hit him with a cluestick [uni-stuttgart.de] on this issue. Yeah, like itojun is despicable and unworthy...

    OTOH, the results are of concern and should be verified by someone less obviously biased. I haven't noticed them in practice on moderately loaded servers though (but I'm biased in the opposite direction).

  • FreeBSD-5!? (Score:3, Insightful)

    by PhlegmMaster (596165) on Sunday October 19 2003, @04:20AM (#7252912)
    I'm wondering, if he was going to be doing a scalability test, why didn't he test the version of FreeBSD that is actually reccomended for production (4.8)?
    He had the time to test the stable and devel versions of the linux kernel, but only the new technology version of freebsd?
    • Re:FreeBSD-5!? by mcbridematt (Score:1) Sunday October 19 2003, @06:07AM
      • Re:FreeBSD-5!? by JanneM (Score:1) Sunday October 19 2003, @07:13AM
        • Re:FreeBSD-5!? by mcbridematt (Score:1) Sunday October 19 2003, @08:52AM
      • Re:FreeBSD-5!? by mcbridematt (Score:1) Monday October 20 2003, @07:41AM
    • Re:FreeBSD-5!? by dutchdabomb (Score:1) Sunday October 19 2003, @01:03PM
    • Re:FreeBSD-5!? by Inoshiro (Score:2) Sunday October 19 2003, @02:53PM
    • 1 reply beneath your current threshold.
  • Hmm Scaling . . (Score:2)

    by OverlordQ (264228) on Sunday October 19 2003, @04:26AM (#7252928)
    (Last Journal: Thursday February 15 2007, @08:00PM)
    Evidently his site doesn't handle traffic very well, especially on the Slashdot Scale.
    • 1 reply beneath your current threshold.
  • by raahul_da_man (469058) on Sunday October 19 2003, @04:33AM (#7252951)
    It's convinced me 2.6 is a *lot* better than 2.4. I'm not really surprised at the results. FreeBSD has never seemed faster than Linux, while Open has seemed noticeably slower.

    Does anyone know why Open underperformed so dramatically? Can the OpenBSD development team comment on this benchmark ?
  • Survive... (Score:1)

    by RealUlli (1365) on Sunday October 19 2003, @04:34AM (#7252952)
    (http://home.pages.de/~RealUlli/)
    The ultimate goal, however, is to demonstrate scalability by surviving a Slashdotting.

    Well, it didn't. ;-)

    Cheers, Ulli

    • 1 reply beneath your current threshold.
  • From /usr/src/UPDATING:

    NOTE TO PEOPLE WHO THINK THAT 5.0-CURRENT IS SLOW:
    FreeBSD 5.0-CURRENT has many debugging features turned on, in
    both the kernel and userland. These features attempt to detect
    incorrect use of system primitives, and encourage loud failure
    through extra sanity checking and fail stop semantics. They
    also substantially impact system performance. If you want to
    do performance measurement, benchmarking, and optimization,
    you'll want to turn them off. This includes various WITNESS-
    related kernel options, INVARIANTS, malloc debugging flags
    in userland, and various verbose features in the kernel. Many
    developers choose to disable these features on build machines
    to maximize performance.

  • O(1) on all (Score:2)

    by dtfinch (661405) * on Sunday October 19 2003, @05:15AM (#7253043)
    (Last Journal: Monday September 25 2006, @01:19PM)
    I knew the Linux 2.6 kernel was supposed to be faster and more scalable, but, damn, that's awesome.
  • Nothing new here (Score:5, Insightful)

    by chrysalis (50680) * on Sunday October 19 2003, @05:20AM (#7253051)
    (http://00f.net/)
    There's no need for such a very technical benchmark.

    Regular usage of various operating systems on the same host makes it obvious.

    When it comes to speed and features (or bloat), Linux is more efficient than FreeBSD, NetBSD and OpenBSD. This is especially significant in SMP environments.

    Linux users are always talking about the just-released experimental patches that will help their system to get 0.1% faster, or the most aggressive flags to optimize their Gentoo system.

    BSD users just advocate their system with the generic word "robust".

    Nowadays, stability is not really the key. Every Linux or BSD free operating system has basically the same stability. The software is the same, with the same bugs. The package system have equivalents (Debian works on NetBSD, Gentoo works a lot like BSD ports, etc) and support for common hardware is almost identical.

    The reason to choose one OS over another is often more political than technical. People tend to use FreeBSD just to try "something else". People tend to use Linux because the Mandrake/RedHat/Conectiva/SuSE installers are beautiful or because Gentoo is fashion and a good way to learn what Unices are made of.

    But if this is just to use common software like Apache and Qmail there's no real difference except speed. If this is what you need, Linux is definitely the best choice nowadays, especially since 2.6 kernels are almost ready for production use.

    For other needs, your mileage may vary.

    For instance I love OpenBSD for development. The compiler and the libc have very handy features to automatically detect bogus code. And the man pages are also excellent, with helpful hints.

    For firewalls and trafic shaping, I wouldn't use anything but *BSD because of PF. PF is really the best thing in *BSD systems IMHO. The firewall is very easy to configure yet extremely powerful and fast. And I was fond of Iptables before.

    For bridging and transparent firewalls, I would also use BSD because it seems to work better than Linux in this area.

    In fact it's just like the girl of your dreams. Everyone's always looking for the perfect operating system that will perfectly fit all needs, but it just doesn't exist.

  • by sir_cello (634395) on Sunday October 19 2003, @06:00AM (#7253122)

    Your scientific method isn't one, and these results are invalid as a result.

    You took in-progress development release of OpenBSD compared to stable releases of other operating systems (even your NetBSD was a RELEASE rather than a CURRENT). CURRENT is always going to be an in progress development and not entirely stable.
  • by astaines (451138) on Sunday October 19 2003, @06:01AM (#7253123)

    Computer benchmarking data are (usually) quite noisy. As such plain graphs, like those shown are not very useful. Simple statistical graphics like lowesss plots are much easier to interpret. Take a look at the for some ideas. [r-project.org]


    This stuff is easy to use, at least if you have a computer background, and would produce far better graphs.

  • Apple marketing? (Score:2)

    by B747SP (179471) <slashdot@selfabusedelephant.com> on Sunday October 19 2003, @06:11AM (#7253142)
    Well, I'm not entirely sure whether these results demonstrate that Theo is an even bigger dickhead than we all already thought he was, or if this guy is a perfect candidate for a position on Apple's G4 benchmarking and marketing team. Maybe both? Do you have to be able to read docs to market G4s?
  • -1 troll story (Score:1, Troll)

    by SharpFang (651121) on Sunday October 19 2003, @06:36AM (#7253206)
    (http://sharpy.xox.pl/ | Last Journal: Wednesday September 14 2005, @02:12PM)
    From the article:

    The OpenBSD and FreeBSD graphs stop early because OpenBSD crashed when I forked more processes, and I couldn't find out how to increase FreeBSD's system limit on the number of processes (sysctl said the value was read-only).

    MOD THIS STORY DOWN AS A TROLL! BSD NOT DEAD!

  • I read many comments claiming that linux 2.6 is _much_ better than FBSD 5.1 - re-read the article please. I know its hard for a linux zealot not to exaggerate the results, just as it is difficult for a FreeBSD zealot not to get over-defensive.

    As to the quirks. As someone [slashdot.org] pointed out, the author should have read the handbook.

    Also, the FreeBSD kernel (well, the OS, as the kernel isn't developed separately from the rest of the system) development is more in its earlier stages than linux kernel development. There is no code freeze yet (I think) and I very much doubt that the 5.x branch will be stabilized until we see a 5.3 release. Thats not going to happen until spring (5.2 is due in December). In other words, while 2.6 underwent some stages of optimization, this is yet to happen to the 5.x branch. Yes folks, the results you see from FBSD is still that of an unoptimized kernel - and that's not as bad as zealotry (from either side) would have it,no? ;-)
  • by sir_cello (634395) on Sunday October 19 2003, @07:24AM (#7253344)

    It's this type of information that does OSS a disservice - making it look amateur and unprofessional.

    You used an pseudo-scientific method, i.e. your graphs are nice, your data points are nice, but you forgot the fundamentals of testing: you didn't clarify the exact projects you were testing (you should have asked the project leads for advice on which versions of the projects to test against), nor did you asked for feedback and determine the reasons for the anomalous conditions (e.g. the FreeBSD maxprocs could have been resolved by a quick question to a FreeBSD expert). This is a lowbrow technical approach.

    Not only that, you presented the pseudo-science results to a large Linux congress, so that while Linux does perform well (undisputed) and everyone can pat themselves on the back, you make the congress look amateur and unprofessional.

    The result is that many of us that do know a thing or two about methodology, experimentation, engineering, etc, will simply interpret your results as providing a "rough general guideline", but "leaving too many questions unanswered" and continue to think that "a lot of the linux community is still amateur and unprofessional".

  • by udippel (562132) on Sunday October 19 2003, @07:46AM (#7253410)
    I cannot argue on scalability. Benchmarking should be more, though.

    I had been running a SOHO-server on P75, 128MB, Dial-up/DSL, DHCP, Print-server, Apache, SSH, hylafax, Firewall on Linux for some years. Then I tried OpenBSD for a change. It runs with about 1/3 of the memory (Linux around 100 MB, OpenBSD about 35 MB).

    Since I installed it, there were just 8 patches to be added (that is about 1 per month); installed pretty easily. I felt quite safe and the system has been pretty stable.

    Though I don't see too much of a future in it, I appreciate its safety considerations and - in my case - stability.

    My desktops run Linux, but currently I see no contender as replacement for the OpenBSD-box. I wouldn't mind, though, to go back to Linux if there was anything equivalent.

    Let's simply assume that OpenBSD will remain in the niche for small (it will probably never be SMP) servers of security-concious admins with low hardware requirements and low throughput.
    • 1 reply beneath your current threshold.
  • open BSD, slowliness (Score:2, Interesting)

    by lunardude (548375) on Sunday October 19 2003, @09:54AM (#7253875)
    In The article, The author mentions that he finds "unacceptable", or embarassing, a few things about openbsd, mainly concerning it's general speed and scalability.

    ALthought not beeing an expert in bsd nor in linux, maybe i'm rong, but isn't OpenBSD made to be secure, and not the fastest operatin system ( additionally, comparing oragnges to apples, by testing release/current/stable, wasn't the best way of comparing those OSes) ????

    By implementing a few feature in the OpenBSD stack and Kernell, I guess that the devellopers are delibetatelly sacrifying performance in order to get an Os less vulnerable to DOS and other vulnerabilities

    It owuld be interesting to compare the same OSes in a security test to find out which OS is more secure, if speed and scalability isn't the only issue in a OS...
  • by zamurai (457391) on Sunday October 19 2003, @11:51AM (#7254418)
    Felix is more than opinionated about Linux vs. BSD. Check the qmail mailing list archives for his rants with BSD users about filesystem and other performance issues. In particular, he's had a number of run-ins with those on the list who use OpenBSD--the one he labels a `stinker'.
  • by aszaidi (464751) on Sunday October 19 2003, @11:52AM (#7254431)
    (http://sajjadzaidi.com/)
    He doesn't give details about the installation of each OS. Did he benchmark them one by one after wiping out the previous OS? Or did he just partition the disk and put each OS in its separate partition? My impression is that he did the latter though I may be mistaken.

    Any hardware junkie knows that data reads and writes can be upto 2x as fast at the start cylinders of a disk than at the end. If he put FreeBSD in the beginning and OpenBSD at the end, it would really skew the results.

    Why would anyone ask a rhetorical question?
  • First graph (Score:1)

    by Maxwell (13985) on Sunday October 19 2003, @11:55AM (#7254452)
    (http://www.buckethead.org/)
    Conclusion: NetBSD outperforms all other operating systems in this benchmark. However, all contestants scale equally well, there are no clear losers. And the overall latency in this benchmark is so low that the results are interchangeable in practice.

    The first graph, which this quote is under, does not support this. It clearly shows Linux 2.6 using HALF the CPU cycles of NetBSD above 4000 connections. How did he get this conclusion? 2.6 is the clear winner in his tests.

    JON
    • Re:First graph by Fefe (Score:2) Sunday October 19 2003, @12:52PM
    • Re:First graph by donscarletti (Score:2) Sunday October 19 2003, @10:04PM
    • 1 reply beneath your current threshold.
  • Real point: they show improvement (Score:3, Interesting)

    by iabervon (1971) on Sunday October 19 2003, @12:48PM (#7254779)
    (http://iabervon.org/~barkalow/ | Last Journal: Saturday May 31 2003, @02:01AM)
    The interesting point is that all of these operating systems seem to be getting faster. It seems to come down to how many recent developments have been integrated into the version being tested, not any inherent differences between operating systems. This is, of course, as it should be: the source for all of these operating systems is available, and there are even frequently papers describing the techniques. If a technique is, in fact, better, it should eventually be adopted by all of them, and so your results will depend on how much has been adopted in the version you're testing.

    It is encouraging to see that all of these developers are competing with the real opponent, which is not each other or even Microsoft, but the slashdot effect. After all, the goal should not be simply to be better than the others, but to be sufficient for the user's purpose, which is not hampered but rather assisted by sharing all of your tricks. It can sometimes seem like there are endless wars between Linux and BSD, but, behind the scenes, the sides actually share information. Never as much as they'd like, but always more than people think.
  • In reference to a particular metric (not the entire suite of tests) he says "Even Windows would probably outperform OpenBSD"

    Talk about taking the kids gloves off!

    -Adam
    • 1 reply beneath your current threshold.
  • Solaris... (Score:2)

    by ImpTech (549794) on Sunday October 19 2003, @01:07PM (#7254888)
    I know this benchmark is about open source and all, but I would love to see Solaris thrown into the mix. With all its vaunted scalability and stability, I'd love to see what it actually does better. I guess it would have to be the Intel version, but I would think their kernel algorithms should be the same across architectures.
  • Graphing 101 (Score:2)

    by gerardrj (207690) * on Sunday October 19 2003, @01:47PM (#7255139)
    (Last Journal: Friday November 10 2006, @10:38PM)
    I for one had a terribly hard time reading the graphs. Three problems in particular:

    1)
    I (like about 20% of the male population) am partially color deficient.
    The graphs' colors where very hard for me to distinguish in the ledgend; what with the 1 pixel wide font and sample line/symbols. The colors chosen were all of about the same intensity/saturation also, so if you are color blind the entire graph would look all about the same level of gray. This problem is further compounded by all the graphs having different color assigments (Linux 2.6

    2)
    crowding. On a few of the graphs, there were differences in order of magnitude between groupings. Since this dataset was supposed to show the fastest, that would naturally be the most interesting data. You don't take photo finishes of the 7th and 8th finishers in an 8 horse race. Using a logarithmic scale for those charts would make the data easier to read, showing detail in the lower(faster) end and allowing the less significant higher(slower) data to "blend" in the curve.

    3)
    Point size. The symbols are too large for that many data points. This may be personal preference, but single points should be used for scatter graphs like these, not symbols. Symbols may mask other more subtle changes in the results because they overflow other points nearby. Similarly, make your legend samples substantial, a block of color is much more apreciated than a single pixel line, and bold your identifier text.

    If you make graphs for the web, please take them in to Photoshop, GIMP, or something similar and reduce or remove the color data (convert to grey scale). The chart should still be at least fairly reable in greyscale.
    Choose a wide range of Hue, Saturation and Lightness
    If you are going to use light/desturated colors, use a black background. If you will use dark/heavily saturated colors, use a light background.
    If you can't/won't do that, then please try to make the raw data available in CSV or some other "universal" format so we can create our own graphs.
  • hmm (Score:1)

    by kazoosandinstruments (717278) on Sunday October 19 2003, @01:49PM (#7255148)
    Ha, well, i don't know too much about the internals of kernels and OSs and stuff, but hey, why doesn't somebody take the relevant code of the speediest winner of each of these "tests" and smack it all together into one "Performance-enhanced OS"... ha, plz no flames! i understand this would not be a light undertaking, but it is an idea i suppose.
    • 1 reply beneath your current threshold.
  • Disappointed (Score:1)

    by brrrrrrt (628665) on Sunday October 19 2003, @05:00PM (#7256204)
    I read the tests and many of the comments on them.

    I was very curious about the test, but very disappointed with everything after I read it.

    The person who performed the tests was very naive when he said this ought to be the end of flamewars about network performance/scalability among the Linux/*BSD users. His test was just not good enough to be meaningful to anyone in the respect of offering acceptable conclusions.

    A proper test would be

    - Performed on proper hardware. I mean, doing a network performance test on an old laptop with inadequate harddisk size, with four very much server-oriented operating systems, one of which (OpenBSD) couldn't even be installed on the same part of the harddisk as the others! That's like organising a race between a Lamborghini, a Lotus, a Porsche and a Ferrari in downtown New York during rush hour, and disqualifying the Ferrari on the grounds that it is difficult to park backwards.

    - Unbiased. This one was very much biased against OpenBSD, and in favour of Linux/'Leanux', as follows from many of the comments made by the tester.

    - Performed by somebody who knew enough about installing and running all of the OS's involved to run tests on them (the tester seems knowledgeable about Linux, but is totally clueless as far as for instance OpenBSD is concerned.)

    - Performed with a test programme that wasn't developed with a bias toward one of the OS's in the test (the test programme involved was developed on Linux, later 'ported to' *BSD.)

    - Described plain fact, by someone who would be objective and who would avoid showing emotions about the subject, much less a general favour or disgust toward the OS's tested. This is especially important because of the sensitive nature of the test subject. There are so many flamewars already!

    - Described withoud prejudice even /if/ the tester happened to be more familiar or friendly with one of the OS's tested. That would really help improve the value of the test. Unprejudiced==scientifical==professional==a virtue.

    - Carefully giving minute details about the test conditions (hardware, software, test programme details, OS installation details..) This test wasn't remotely accurate, look how it even fails to mention at what time OpenBSD-CURRENT was downloaded. That's crucial information.

    - Compare equally. The test is already invalid because it compares an ancient -STABLE NetBSD with a -CURRENT FreeBSD.

    - Be clear about its subject. The test focuses exclusively on network performance/scalability, and then goes on to praise or totally disqualify the OS's tested solely on this ground. The test doesn't look at crucial aspects like security, maintainability, documentation, correctness of design, etcetera, but will still not hesitate to draw very un-subtle conclusions.

    - Ask those who are in the know for comments on the test conclusions before making the results public.
    In this precise case, it would have prevented several stupid factual mistakes from being published (the OpenBSD installation problem and IPv6 idiosyncrasies for instance.)

    - Call into the test all relevant players, or at least represent different groups properly.
    Much as I dislike it - I'm an open source adept as well - Windows is used as a server system on the public internet by many organisations. It should be compared with the other systems in this test, and be given a fair chance. I wouldn't have been surprised if it performed very well, seeing that the TCP/IP stack of modern Windows versions has been largely copied from FreeBSD..
    Also I would have liked it if there were at least one proprietary Unix system (such as Solaris) in the test. Just for the sake of the comparison.

    - Look closer into the reasons, backgrounds, pros and cons of faults that the tests find. Again, if the tester had done this, he would have found that some of the badness he found wasn't a design mistake, but a design decision based on healthily made trade-offs (security trade-offs in t
  • by mistered (28404) on Sunday October 19 2003, @06:40PM (#7256751)
    I tried building this on FreeBSD 4.7 but got an error:

    gcc -o pthreadbench pthreadbench.o -g -L../libowfat/ -lowfat -lpthread
    /usr/libexec/elf/ld: cannot find -lpthread
    *** Error code 1

    Some Google searching turned up a thread discussing this, and as suggested there I replaced -lpthread with -pthread in the Makefile. It built fine after that.

  • by slave 6742 (703775) on Monday October 20 2003, @02:59AM (#7258744)
    It is disappointing to see many of the comments here. Some are good and beneficial comments, others are not.

    Where is the spirit of Sharing Ideas here?

    #Define Flame On

    I suggest that some here Grow Up! Zip your pants up, and quit the p*ss*ng contest. Your 15 seconds of fame are over! Now start acting like team players in this world!

    #Define Flame Off

    How about several of the people here that apparently seem that they know their stuff the best (and they may), smash heads together to come up with some benchmarks that test appropriately each of the different Operating Systems. I do mean all of the different OSs, such as OS/2, HPUX, Solaris 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, Windows (3.1, 95, 98, 2000, NT, etc....) Linux 2.1, 2.2, 2.4, etc... FreeBSD 4.8, 4.9, OpenBSD, NetBSD, etc.....

    I would suggest the following:

    1. Get some people together to come up with a general test that measures specific areas that the kernel is programmed for. These would be tests that would be subject to each OS developers approval (ie, that they test appropriately the area specified). Also, acquire each OS tuning parameters for each specific test.

    2. Use these tests to benchmark where an OS has issues.

    3. Forward these tests to developers who care about the OS they program, to use these as starting points where code may be improved.

    4. Post the tests for each OS. Both before and after fixes.

    Do I know what I am talking about? Probably not, but I was a test engineer for several years and we did perform quite a few test just like Fefe did to see what would happen. As time went on, yes we came up with better and better tests. Accept this fellows test as for what it is, a start of people getting together to put together better and better tests. By the way, he did a fairly good job for one that has gone out of their way to help the nerds out there think about things, some apparently think, while others just like to /. .

    Which OS do I use? Which ever is handy at the moment! Sometimes we don't get a choice!

  • by hegemon17 (702622) on Monday October 20 2003, @09:45AM (#7260308)
    Redoing all those benchmarks without the strange libraries that the author uses and instead using plain syscalls shows that what the benchmarks measure is how the benchmark code performs, not the actual syscalls.

    A real bind benchmarks has a O(1) time on OpenBSD and NetBSD.

    The connect latency benchmark seems seriously confused. And since I don't find any source for it, I must conclude (based on the rest of the benchmarks) that it must be flawed in some way.

    The "read one byte from each page" benchmark on OpenBSD performs that badly because the file used doesn't fit in the buffer cache.OpenBSD doesn't use all the memory for the file cache. When the file fits in the buffer cache OpenBSD has a performance comparable to linux 2.4. Anyone with half a clue about how a computer works would have listened to the disk and noticed that it's making noise.

    And the rants about ipv6 only show that he doesn't know how to write proper ipv6 code and instead relies on a compatibility interfaces that were created for lazy programmers to allow for quick conversion of old ipv4 programs into ipv6.

    The author is clueless and the benchmarks more often measure the performance of the benchmark itself than the operating system.
    • 1 reply beneath your current threshold.
  • by lanswitch (705539) on Sunday October 19 2003, @03:28AM (#7252779)
    Dass ist doch wahnsinn, du Sau! Die leute hier koennen das gar nicht lesen! Kannst du das uebersetzen?
    [ Parent ]
  • Its all dutch (well, german) to me
    [ Parent ]
  • by Ianoo (711633) on Sunday October 19 2003, @03:36AM (#7252801)
    (Last Journal: Friday May 21 2004, @10:08AM)
    Horray for Haxalot! Oh, and the PDF is quite interesting, too.
    [ Parent ]
    • 1 reply beneath your current threshold.
  • by Ianoo (711633) on Sunday October 19 2003, @04:37AM (#7252962)
    (Last Journal: Friday May 21 2004, @10:08AM)
    Alternatively: buy a Mac! Pretend your beloved operating system is now one of the most widespread consumer *nixes! Because we all know that Aqua has nothing to do with it, Mac is BSD!
    [ Parent ]
  • I'M A GERMANIZED HONEY BEE!!! (Score:1, Offtopic)

    Arbeit macht frei!
    [ Parent ]
    • 1 reply beneath your current threshold.
  • by surprise_audit (575743) on Sunday October 19 2003, @10:47AM (#7254093)
    By your reasoning, no benchmarks should ever be done. Don't bother benchmarking Intel and AMD cpus, don't bother benchmarking different kinds of memory, different video cards, different printers. Don't bother comparing cars, TVs, washing machines, &c from different manufacturers, because they're so dissimilar.

    Oh, wait, but we are comparing those dissimilar products when they're all attempting to do the same thing. Why else would car makers boast about engine horsepower, acceleration, traction and braking, for example?

    I guess you didn't actually read the article, where the guy states that he's doing the same damn thing on each OS, using the same hardware, so the only variable is the OS/distro being used. His choice of OS/distro may be suspect, but the results still stand. He's comparing oranges to oranges and telling you which is sweeter, which is juicier, etc.

    [ Parent ]
  • Re:*BSD Problems (Score:1)

    by hdw (564237) on Sunday October 19 2003, @11:39AM (#7254362)
    raider:videos {107} du -sk apa.mpg

    709496 apa.mpg
    raider:videos {109} time cp apa.mpg /slask/
    0.007u 6.593s 0:50.42 13.0% 0+0k 10933+77720io 0pf+0w
    Some 13 MB/s sustained, OpenBSD 3.3 Duron 1200.

    // hdw
    [ Parent ]
  • Re:Nothing but... (Score:1)

    by clifgriffin (676199) on Sunday October 19 2003, @01:17PM (#7254945)
    (http://clifgriffin.com/)
    He speaks truth!!!!
    [ Parent ]
    • 1 reply beneath your current threshold.
  • Re:Golly! (Score:1)

    by metalix (259636) on Sunday October 19 2003, @09:39PM (#7257638)
    Yep. Welcome to slashdot, where anything that shows linux as not perfect OS gets torn apart.
    [ Parent ]
    • 1 reply beneath your current threshold.
  • 43 replies beneath your current threshold.