Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
Programming Operating Systems Software Sun Microsystems Windows IT Technology BSD Linux

Code Quality In Open and Closed Source Kernels 252

Diomidis Spinellis writes "Earlier today I presented at the 30th International Conference on Software Engineering a research paper comparing the code quality of Linux, Windows (its research kernel distribution), OpenSolaris, and FreeBSD. For the comparison I parsed multiple configurations of these systems (more than ten million lines) and stored the results in four databases, where I could run SQL queries on them. This amounted to 8GB of data, 160 million records. (I've made the databases and the SQL queries available online.) The areas I examined were file organization, code structure, code style, preprocessing, and data organization. To my surprise there was no clear winner or loser, but there were interesting differences in specific areas. As the summary concludes: '..the structure and internal quality attributes of a working, non-trivial software artifact will represent first and foremost the engineering requirements of its construction, with the influence of process being marginal, if any.'"
This discussion has been archived. No new comments can be posted.

Code Quality In Open and Closed Source Kernels

Comments Filter:
  • Is it just me? (Score:5, Interesting)

    by Abreu ( 173023 ) on Friday May 16, 2008 @11:55AM (#23434796)
    Or the summary is completely incomprehensible?

    Of course, I could try to RTFA, but hey, this is Slashdot, after all...
    • Re:Is it just me? (Score:5, Insightful)

      by Anonymous Coward on Friday May 16, 2008 @11:58AM (#23434836)
      That is if you can figure out which of the 12 links are the actual FA and which are supporting material.
      • Re:Is it just me? (Score:5, Interesting)

        by utopianfiat ( 774016 ) on Friday May 16, 2008 @02:34PM (#23437810) Journal
        Well, it's the second link. At any rate, the highlights of the data are that for the most part the kernels are tied in the important material, except:

        % style conforming lines: FBSD:77.27 LIN:77.96 SOLARIS:84.32 WIN:33.30
        % style conforming typedef identifiers: FBSD:57.1 LIN:59.2 SOLARIS:86.9 WIN:100.0
        % style conforming aggregate tags: FBSD:0.0 LIN:0.0 SOLARIS:20.7 WIN:98.2

        (I'm far too lazy to clean up the rest)

        % of variable declarations with global scope 0.36 0.19 1.02 1.86
        % of variable operands with global scope 3.3 0.5 1.3 2.3
        % of identifiers with wrongly global scope 0.28 0.17 1.51 3.53
    • Re:Is it just me? (Score:4, Interesting)

      by Bazman ( 4849 ) on Friday May 16, 2008 @12:12PM (#23435076) Journal
      Well, it's not just you, but probably millions like you. Plenty of the summary is comprehensible, but I get the fear that it's really just a slashvertisement for his book (third link in summary).

      • Re: (Score:2, Informative)

        by stavrosg ( 893274 )
        TFA is the second link, but yes, the summary does not do much to help you figure it out.
    • Re:Is it just me? (Score:5, Insightful)

      by raddan ( 519638 ) on Friday May 16, 2008 @12:18PM (#23435184)
      It's not a very good summary, but the paper is well-written, which is interesting considering that the author is the one who submitted the summary to Slashdot. I suspect that he assumes we have more familiarity with the subject than we actually do.
    • Re: (Score:2, Insightful)

      I'll attempt a radical paraphrase: "Form follows function, not process."
    • Re:Is it just me? (Score:5, Insightful)

      by Diomidis Spinellis ( 661697 ) on Friday May 16, 2008 @12:50PM (#23435794) Homepage
      I didn't write the last part when I submitted the story, and, yes, the summary given here is comprehensible, because it appears out of context. What the sentence '..the structure and internal quality attributes of a working, non-trivial software artifact will represent first and foremost the engineering requirements of its construction, with the influence of process being marginal, if any.' means is that when you build something complex and demanding, say a dam or an operating system kernel, the end result will have a specific level of quality, no matter how you build it. For this reason the differences in the software built with a tightly-controlled proprietary software process and that built using an open-source process are not that big.
      • Re:Is it just me? (Score:5, Insightful)

        by legutierr ( 1199887 ) on Friday May 16, 2008 @01:37PM (#23436570)
        How useful is it to write something about computers that needs to be translated for the slashdot audience? Jargon is a great way to provide specialized information to insiders quickly and efficiently, but this is slashdot. If slashdot readers need for you to restate your description of a problem or observation related to the Linux kernel (even if that description is taken out of context), could it be that the paper could be written in a more open manner? The quote you provided from your paper seems to speak to a narrow audience; how narrow must your audience be, however, if it excludes a good portion of slashdot's readers?

        If I seem overly critical, I do not mean to, it is only that I hate to see good, useful research made less accessible to non-academics by the use of academic language.
        • Re: (Score:3, Insightful)

          This is a very good point...
        • Re: (Score:3, Insightful)

          by Allador ( 537449 )
          I think you're vastly overestimating the overall knowledge, experience, and education in this context of the /. readership.

          Only a small portion of the /. readers are programmers/developers (or in an academic speciality in this field).

          Even amongst that population, only a small percentage are qualified in large operating systems development.

          Similarly, a small percentage of the programmer subpopulation of /. will be familiar with the various theory, metrics and approaches to measuring and analyzing code qualit
      • Re: (Score:3, Insightful)

        by CherniyVolk ( 513591 )
        ... means is that when you build something complex and demanding, say a dam or an operating system kernel, the end result will have a specific level of quality, no matter how you build it.

        First, Mr. Spinellis I found the report to be rather intriguing and captivating. I much respect the work put into it, and I think it'll prove valuable resource for study or reference.

        With that said, the above quote struck a chord with me.

        Let's take fire control systems for weapons of mass destruction. Without going into
        • Re:Is it just me? (Score:4, Informative)

          by Diomidis Spinellis ( 661697 ) on Saturday May 17, 2008 @04:03AM (#23444082) Homepage
          Please let me clarify here:

          I can not extrapolate the agreeable portions of your thought to the seemingly obvious short comings of the Windows operating system. On any facet, whether it is security, stability, functionality or reliability. Windows is, far behind on all fronts.... aside from secrecy from a Microsoft point of view.
          I'm not claiming anything regarding these external quality attributes of Windows, the metrics I collected just show that there are no vast differences in the code's quality.

          Or, perhaps, the WRK has been a meticulous focus at Microsoft before it's release... this is likely possible, as it's WIDELY known, from nearly ALL examples of closed source proprietary software being released to the Open Source, that it takes years just to clean up and prepare for the ultra high standards of the OS community.
          This is entirely possible. In fact, a README file in the distribution states:

          The primary modifications to WRK from the released kernel are related to cleanup and removal of server support, such as code related to the Intel IA64.
  • by Number6.2 ( 71553 ) * on Friday May 16, 2008 @12:03PM (#23434922) Homepage Journal
    That you have neither capitalized on your shared synergies, nor have you recovered your cherished paradigms.

    Oh. Wait. This is about propeller-head stuff rather than management stuff. Lemme get my "Handbook of postmodern buzz words"...
  • Not that surprising (Score:5, Interesting)

    by abigor ( 540274 ) on Friday May 16, 2008 @12:05PM (#23434958)
    Final line in the paper: "Therefore, the most we can read from the overall balance of marks is that open source development approaches do not produce software of markedly higher quality than proprietary software development."

    Interesting, but not shocking for those who have worked with disciplined commercial teams. I wonder what the results would be in less critical areas than the kernel, say certain types of applications.
    • by ivan256 ( 17499 ) on Friday May 16, 2008 @12:25PM (#23435320)
      It's obvious what the results would be.

      Half completed, unpolished commercial software usually stays unreleased and safe from this sort of scrutiny. However many of the same types of projects get left out in the open and easily visible to everybody when developed as open source. The low code quality of these projects would drag down the average for open source projects as a whole.

      On the lighter side, you could say that you'd only consider software that was "out of beta" or version 1.0 or greater, but that would leave out most open source projects and commercial "Web 2.0" products....
      • Re: (Score:3, Insightful)

        by xenocide2 ( 231786 )

        On the lighter side, you could say that you'd only consider software that was "out of beta" or version 1.0 or greater, but that would leave out most open source projects and commercial "Web 2.0" products....

        Then restrain yourself to "what Fedora ships" or "what Canonical supports in main". These are the presumably viable software products with a living upstream.

        But you missed an interesting problem: failed commercial programs sometimes convert into open source projects. Its not clear to me whether this is a

    • Re: (Score:3, Insightful)

      So open source software is not of 'markedly' higher quality. If it is of even 'slightly' higher quality, or 'exactly the same quality' as closed source software, then the fact that it costs less, and gives users freedoms that they don't have with closed source software, means that closed source is doomed.
      • Re: (Score:3, Interesting)

        by abigor ( 540274 )
        Well, not necessarily. Perhaps for certain types of commodity applications, like office suites, but even then, it's tough to say. That's why I was interested in the comparison. Your assertion is certainly not true for games, for example.

        Generally speaking, commercial desktop apps are still way ahead of their open counterparts, with the exception of code development tools and anything that directly implements a standard (browsers, mail clients, etc.)

        One reason for this is that code quality as measured in th
        • by FishWithAHammer ( 957772 ) on Friday May 16, 2008 @01:34PM (#23436528)

          Generally speaking, commercial desktop apps are still way ahead of their open counterparts, with the exception of code development tools and anything that directly implements a standard (browsers, mail clients, etc.)
          Code development tools? VS says hi. (And somebody is now going to leap in and say that that monstrosity Eclipse is somehow "better" than VS...this will be amusing.)
          • by abigor ( 540274 )
            VS is great, don't get me wrong. I meant sort of in general, beyond just MS development, then the free/open tools are really good. Of course, I am using IDEA at the moment, which is not open, so I guess I'm not exactly practising what I preach.
          • by phantomfive ( 622387 ) on Friday May 16, 2008 @08:08PM (#23441744) Journal
            As I take a break from VS to enjoy some time on slashdot, I have no understanding of why people think VS is so amazing.

            Is it because of intellisense? That's kind of nice, especially when your code is so disorganized that you don't remember where stuff is defined. Or if you can't open stuff up in two different windows to see where it is defined, like VS prevents you from doing (yeah, you sort of can, but it's stuck in the main VS window).

            Is it because of the debugger? Sure, the debugger is nice, and I like it, but it only helps get rid of the easy bugs. The bugs that really eat your development time are the ones that only manifest themselves after the program has been running for a few hours/days, and usually a debugger doesn't help much with that. Besides, every other IDE comes with a debugger, even GDB works fine if you can handle arcane keystroke combinations.

            And on top of it, Solutions and projects in VS are horrible. Why does VS try to save the solution every time I quit? Makefiles have some awful syntax, but at least when they change, I know it's because of what I've done, and I know how to fix it.

            That said, I don't consider VS to be a bad IDE, it is reasonably decent. I just don't understand the logic of these guys who think that VS is the greatest IDE ever. It's a question, not a flame.
      • Re: (Score:3, Insightful)

        So open source software is not of 'markedly' higher quality. If it is of even 'slightly' higher quality, or 'exactly the same quality' as closed source software, then the fact that it costs less, and gives users freedoms that they don't have with closed source software, means that closed source is doomed.

        The problem with your conclusion is that it assumes that code quality as measured in this sort of way is the only or even the most important thing.

        It's like buying clothes. Sure, the quality of the clothes
      • by Diomidis Spinellis ( 661697 ) on Friday May 16, 2008 @03:36PM (#23438752) Homepage
        I don't think that my results can support us in making arguments regarding 'slightly' higher quality, or 'exactly the same quality'. My figures are based on possibly interdependent, unweighted, and unvalidated metrics. Therefore they only allow us to make conclusions involving large differences.
    • Interesting, but not shocking

      Considering that few open source developers are strictly open source, that's hardly a surprise. I'd be willing to bet many open source developers are also part of disciplined commercial teams.

      The flip side of that coin is just as intriguing. Open source development models don't produce software of notably inferior quality either. That should send a shivey through Castle Redmondore.

    • by hey! ( 33014 )
      Well, for a certain definition of "quality".

      Maybe a better way of saying this is that open source programmers aren't better programmers than closed source ones.

      But nobody ever said open source programmers are better. The argument is that open source software gets continually better from a user's perspective. If it doesn't for enough users, somebody else gives them what they want. If you aren't happy with SUSE's direction, you can go to RHEL and vice versa without creating a lot of fuss. Chances are some
    • by jd ( 1658 )
      OpenSolaris is hybrid open/closed. Solaris was originally closed, so the style and quality control will still be at least partially along closed lines, but is now open so will contain some style and quality control aspects of open source. Without knowing the ratio, however, it is not a particularly useful kernel to study. It is most useful as a sanity check, as it must be somewhere between the two styles, but since you cannot define where between, it has no value in the initial analysis.

      This leaves just o

    • Re: (Score:3, Interesting)

      by raddan ( 519638 )
      But the author also points out that his measure of "quality" is limited. From the paper:

      Other methodological limitations of this study are the small number of (admittedly large and important) systems studied, the language specificity of the employed metrics, and the coverage of only maintainability and portability from the space of all software quality attributes. This last limitation means that the study fails to take into account the large and important set of quality attributes that are typically determined at runtime: functionality, reliability, usability, and efficiency. However, these missing attributes are affected by configuration, tuning, and workload selection. Studying them would introduce additional subjective criteria. The controversy surrounding studies comparing competing operating systems in areas like security or performance demonstrates the difficulty of such approaches.

      From the end-user perspective functionality, reliability, usability, and efficiency are pretty much the entire thing. Most users couldn't care less that a piece of software is hard to maintain as long as it does what he wants reliably, consistently, and with a minimal amount of cognitive load. So this paper is aimed more at applying traditional software engineering metrics to four pieces of real-world software. The

    • I wonder what the results would be...

      I wonder what the results would be if a real, commercial Windows kernel were tested instead of a research-oriented one. I doubt if the research kernel was ever subjected to the "have to release it on time" programming deadlines.

  • by wellingtonsteve ( 892855 ) <wellingtonsteve AT gmail DOT com> on Friday May 16, 2008 @12:05PM (#23434962)
    ..that Open Source code is of quality, but at least the point of things like the GPL is that you have the power to change that, and improve that code..
    • by KutuluWare ( 791333 ) <kutulu.kutulu@org> on Friday May 16, 2008 @04:45PM (#23439718) Homepage
      You haven't been paying attention to many Open Source proponents if you haven't ever seen them claim that Open Source code is of vastly superior quality than proprietary. Hell, ESR's claim to fame is a whole paper he wrote on that exact topic. For example, the OSI itself puts this claim at the very top of their advocacy document on selling OSS to your management:

      The foundation of the business case for open-source is high reliability. Open-source software is peer-reviewed software; it is more reliable than closed, proprietary software. Mature open-source code is as bulletproof as software ever gets.
        Open Source Case for Business [opensource.org]
      There is a pretty clear divide in the F/OSS community between the OSI-side-people, who view Open Source as a development model that leads to better software with fewer bugs and quicker turnaround; and the FSF-side-people who think of Free Software as a moral imperative that leads to more freedom in addition to better software with fewer bugs and quicker turnaround.

      Having worked heavily in both areas of software development, I think this particular article's conclusion was obvious: code quality depends on the people who wrote it, not the process the used to license it. But only people who have done extensive proprietary and open-source development could really see that first-hand, and our opinions are automatically dismissed as being pro-Microsoft shills. Thus, I predict this paper will be roasted over an open flame, crushed into a tiny ball, soaked in gasoline, lit on fire, and ejected into deep space by the most devoted open source proponents in both camps.
  • CScout Compilation (Score:5, Insightful)

    by allenw ( 33234 ) on Friday May 16, 2008 @12:08PM (#23435028) Homepage Journal
    "The OpenSolaris kernel was a welcomed surprise: it was the only body of source code that did not require any extensions to CScout in order to compile."

    Given that the Solaris kernel has been compiled by two very different compilers (Sun Studio, of course, and gcc), it isn't that surprising. Because of the compiler issues, it is likely the most ANSI compliant of the bunch.
  • by davejenkins ( 99111 ) <slashdot@NOSPam.davejenkins.com> on Friday May 16, 2008 @12:09PM (#23435034) Homepage
    If I am understanding correctly, you were looking for 'winners' and 'losers' (weasel words in and of themselves, but anyway...) in terms of 'quality' (another semi-subjective term that could make someone go crazy and drive motorcycles across the country for the rest of their lives).

    You found that '..the structure and internal quality attributes of a working, non-trivial software artifact will represent first and foremost the engineering requirements of its construction, with the influence of process being marginal, if any.' -- or in plain English: "the app specs had a much bigger influence when compared to internal efficiencies".

    I would wonder if you're just seeing a statistical wash-out. Are you dealing with data sets (tens of millions of lines and thousands of functions) that are so large, that patterns simply get washed out in the analysis?

    Oh dear, my post is no more clear than the summary...
    • Re: (Score:3, Interesting)

      in plain English: "the app specs had a much bigger influence when compared to internal efficiencies".

      It sounds more like they're saying "If someone built it, and someone else is using it, and it's important, then the code quality is going to be pretty good. If it matters, it's going to get attention and be improved."

      Of course, I can think of a bunch of counter-examples in Windows where something was important *to me* and mattered *to me* and no one at Microsoft saw fit to do anything about it for decad

      • by drsmithy ( 35869 )

        Of course, I can think of a bunch of counter-examples in Windows where something was important *to me* and mattered *to me* and no one at Microsoft saw fit to do anything about it for decades.

        For example ?

        • OK, sure that's fair. Here's one:

          When copying/move a bunch of files, if Windows encounters an error that prevents a file from being read/written, the entire copy aborts at the point where the problem occurred.

          I'd love Windows to automatically fail the problem file while continuing to try to copy/move the remaining files, then present you with some kind of error report at the end saying "These files couldn't be copied for [reason]."

          There's file management utilities that you can use to get around t

    • "Oh dear, my post is no more clear than the summary..."

      For those of us who have read this [wikipedia.org], it is ;)
    • I would wonder if you're just seeing a statistical wash-out. Are you dealing with data sets (tens of millions of lines and thousands of functions) that are so large, that patterns simply get washed out in the analysis?
      I don't think so. I can't now do an analysis for a counterexample, but I am pretty sure that if I run the same metrics on, say, the bottom 20% in terms of downloads, of Sourceforge projects I will very different results.
    • by raddan ( 519638 ) on Friday May 16, 2008 @01:37PM (#23436568)
      With regard to the guy who went crazy and drove his motorcycle across the country-- I think the point of the book was to demonstrate that "subjective" and "objective" are specious terms. Science gets all hot and bothered when words like "good" and "bad" are used, but not when words like "point" are used. So if we can make allowances for axiomatic terms, why not so-called "qualitative" terms? After all, the word "axiom" means, according to Wikipedia:

      The word "axiom" comes from the Greek word axioma a verbal noun from the verb axioein, meaning "to deem worthy", but also "to require", which in turn comes from axios, meaning "being in balance", and hence "having (the same) value (as)", "worthy", "proper". Among the ancient Greek philosophers an axiom was a claim which could be seen to be true without any need for proof.
      Indeed, if you look at many of our "quantitative" measures, they are, at their heart, a formalization of "goodness" and "badness". If you're a mathematician, you might argue that this is not true (since there are loads of mathematical constructs whose only requirement is simply self-consistency and not some conformance to any external phenomenon), but if you're an engineer, you're whole career balances on the fine points of "goodness" and "badness". It is an essential concept!

      My personal opinion is that if statistics are a wash-out in general, then the researcher is asking the wrong questions. I know that the author pre-defined his metrics in order to avoid bias, but that's not necessarily good science. Scientific questions should be directed toward answering specific questions, and the investigatory process must allow the scientist to ask new questions based on new data.

      There is clear non-anecdotal evidence that these operating systems behave differently (and, additionally, we assign a qualitative meaning to this behavior), so the question as I understand it is: is this a result of the development style of the OS programmers? The author should seek to answer that question as unambiguously as possible. If the answer to that question is "it is unclear", then the author should have gone back and asked more questions before he published his paper, because all he has shown is that the investigatory techniques he used are ill-suited to answering the question he posed.
  • Really? (Score:3, Insightful)

    by jastus ( 996055 ) on Friday May 16, 2008 @12:19PM (#23435200)
    I'm sorry, but if this is what passes for serious academic computer-science work, close the schools. This all appears to boil down to: quality code (definition left to the reader) is produced by good programmers (can't define, but I know one when I see his/her code) who are given the time to produce quality code. Rushed projects by teams of average-to-crappy programmers results in low-quality code. All the tools and management theories in the world have little impact on this basic fact of life. My PhD, please?
    • Forget the PhD, if you have really learned that lesson, then you need to be made CIO at any one of my former employers.
    • Re: (Score:3, Insightful)

      by jjohnson ( 62583 )
      If you'd RTFA, you'd know that there's a lot more to what the author said than that. He says nothing about a relationship between the quality of programmers and the quality of code; he says nothing about the time taken to develop code, and makes no conclusions about its effect on code quality.

      What he says is that a cluster of metrics that collectively say something general about code quality (e.g., better code tends to have smaller files with fewer LOC; worse code has more global functions and namespace po
  • The 99% Solution (Score:5, Interesting)

    by SuperKendall ( 25149 ) on Friday May 16, 2008 @12:21PM (#23435232)
    So while looking at the data collected, I had to wonder if some of the conclusions reached were not something of a matter of weighting - I saw some things pretty troubling about the WRK. Among the top of my list was a 99.8% global function count!!!

    This would explain some things like lower LOC count - after all, if you just have a bunch of global functions there's no need for a lot of API wrapping, you just call away.

    I do hate to lean on LOC as any kind of metric but - even besides that, the far lower count of Windows made me wonder how much there, is there. Is the Windows kernel so much tighter or is it just doing less? That one metric would seem to make further conclusions hard to reach since it's such a different style.

    Also, on a side note I would say another conclusion you could reach is that open source would tend to be more readable, with the WRK having a 33.30% adherence to code style and the others being 77-83%. That meshes with my experience working on corporate code, where over time coding styles change on more of a whim whereas in an open source project, it's more important to keep a common look to the code for maintainability. (That's important for corporate code too - it's just that there's usually no-one assigned to care about that).

    • Re: (Score:2, Insightful)

      by Yosi ( 139306 )
      The piece of Windows they had did not include drivers. It says:

      Excluded from the kernel code are the device drivers, and the plug-and-play, power management, and virtual DOS subsystems. The missing parts explain the large size difference between the WRK and the other three kernels.


      Much of the code in Linux, for instance, is drivers.
    • Re:The 99% Solution (Score:4, Informative)

      by Diomidis Spinellis ( 661697 ) on Friday May 16, 2008 @01:03PM (#23436024) Homepage

      So while looking at the data collected, I had to wonder if some of the conclusions reached were not something of a matter of weighting - I saw some things pretty troubling about the WRK. Among the top of my list was a 99.8% global function count!!!
      I guess Microsoft uses a non-C linker-specific mechanism to isolate their functions, for instance by linking their code into modules. But yes, this is a troubling number.

      This would explain some things like lower LOC count - after all, if you just have a bunch of global functions there's no need for a lot of API wrapping, you just call away.
      The lower LOC comes from the fact that WRK is s subset of Windows. It does not include device drivers, and the plug-and-play, power management, and virtual DOS subsystems.

      Also, on a side note I would say another conclusion you could reach is that open source would tend to be more readable, with the WRK having a 33.30% adherence to code style and the others being 77-83%. That meshes with my experience working on corporate code, where over time coding styles change on more of a whim whereas in an open source project, it's more important to keep a common look to the code for maintainability. (That's important for corporate code too - it's just that there's usually no-one assigned to care about that).
      About 15 years ago I chanced upon code in a device driver that Microsoft distributed with something like a DDK that had comments written in Spanish. The situation in WRK is markedly better, but keep in mind that Microsoft distributes WRK for research and teaching.
  • KLOCs? (Score:5, Insightful)

    by Baavgai ( 598847 ) on Friday May 16, 2008 @12:28PM (#23435380) Homepage
    If good code and bad code were a simple automated analysis away, don't you think everyone would be doing it? What methodolgy could possibly give a quantitative weighting for "quality"?

    "To my surprise there was no clear winner or loser..." Not really a surprise at all, actually.

    • Re: (Score:2, Funny)

      by getto man d ( 619850 )
      Exactly. Automation excludes creativity. If it were that easy I could just do the following:

      writeGoodCode(int numberOfLines, float ouncesOfCoffeeConsumed)
    • Re: (Score:3, Insightful)

      You can automatically recognize some bad smells of poor quality code. However, this will still let through poor quality code that has been explicitly written to guard against the bad smells. So, you can say for sure that some code stinks, but you can't (and I suspect you will never be able to) tell that some code excels.
  • by abolitiontheory ( 1138999 ) on Friday May 16, 2008 @12:35PM (#23435518)
    Does anybody see that these results are in still favor of open source? The fact is, it's actually a beautiful thing that the difference in quality is marginal. This equality then becomes the rubric by which to judge other elements of the design process, and choices about whether to develop and deploy programs with open source or closed source.

    People make claims about the need for closed source all the time, usually revolving around the need to a predictable level of quality, or some other factor. The fact is, this results proves that its a wash whether you choose open or closed--so why not choose open?

    There's a deep significance here I'm failing to capture completely. Someone else word it better if they can. But there didn't need to be some blow-out victory of open source over closed source for this to be a victory. All open source needed to do was compare--which it did, clearly--with closed source, in terms of value, to secure its worth.

    • This is a very interesting comment. I had not thought of my results in this light, because, based on my experience as a (minor) FreeBSD committer and as a Windows user, I was expecting to see a large difference in favor of open source code. Yes, in the way you put it, open source is a winner.
  • So.... (Score:3, Interesting)

    by jellomizer ( 103300 ) on Friday May 16, 2008 @12:38PM (#23435574)
    The way you choose to license your software doesn't coralate with software quality... Seems logical to me. As how you license your software has very little to do about the code inside the OS.

    Closed Source Developer: I will try to do my best job as I possibly can so I can keep my job and make money because that is what I value.

    Open Source Developer: I will try to do my best job as I possibly can so I can help the comunity and feel better about myself/get myself noticed in the comunity/Something cool to put on my resume... because that is what I value.

    People who choose to license their software OpenSource vs. Closed Source says nothing about their programming ability. There are a bunch of really crappy GNU projects out there as well as a bunch of crappy closed source projects... Yea there is the argument of millions of eyes fixing problems but really when you get millions of people looking at the same thing you will get good and bad ideas so the more good ideas you get the more bad ideas you get and the more people involved the harder it gets to weed out good ones and bad ones. Closed source is effected often by a narrow level of control where bad ideas can be mandated.... All in all everything really ballances out and the effects of the license are negledgeable.
    • Re: (Score:3, Interesting)

      The way you license code can't directly affect its quality, but the way you develop it can. Here are some possible ways in which a company can affect (positively or negatively) the quality of the software:
      • Have managers and an oversight group control quality (+)
      • Through its bureaucracy remove incentives to find creative solutions to quality problems (-)
      • Pay for developers to attend training courses (+)
      • Provide a nice environment free of distractions that allows developers to focus on developing qualit
  • by SixDimensionalArray ( 604334 ) on Friday May 16, 2008 @12:43PM (#23435680)
    I haven't seen anybody else comment on the fact that the statement that the quality of the code had more to do with the engineering than the process through which the code was developed is quite interesting.

    From my personal experiences, it typically seems code is written to solve a specific need. Said another way, in the pursuit of solving a given problem, whatever engineering is required to solve the problem must be accomplished - if existing solutions to problems can be recognized, they can be used (for example, Gang of Four/GOF patterns), otherwise, the problem must have a new solution engineered.

    Seeing as how there are teams successfully developing projects (with both good, and bad code quality) using traditional OO/UML modeling, the software development life-cycle, capability maturity model, scrum, agile, XP/pair programming, and a myriad of other methods, it would seem to be that what the author is saying is, it didn't necessarily matter which method was used, it was how the solution was actually built (the.. robustness of the engineering) that mattered.

    Further clarification on the difference between engineering and "process" would strengthen this paper.

    I went to a Microsoft user group event some time ago - and the presenter described what they believed the process of development of code quality looked like. They suggested the progression of code quality was something like:
    crap -> slightly less crappy -> decent quality -> elegant code.

    Sometimes, your first solution at a given problem is elegant.. sometimes, it's just crap.

    Anyways, just my two cents. Maybe two cents too many.. ;)

    SixD
    • by T.E.D. ( 34228 )

      crap -> slightly less crappy -> decent quality -> elegant code.

      That's only the first half of the life-cycle. The rest would be along the lines of

      -> elegant code with a few special case hacks -> special case hacks with a few lines of elegant code -> crap.
  • "Linux excels in various code structure metrics, but lags in code style. This could be attributed to the work of brilliant motivated programmers who aren't however efficiently managed to pay attention to the details of style. In contrast, the high marks of WRK in code style and low marks in code structure could be attributed to the opposite effect: programmers who are efficiently micro-managed to care about the details of style, but are not given sufficient creative freedom to structure their code in an app
  • Stupid metrics (Score:4, Interesting)

    by Animats ( 122034 ) on Friday May 16, 2008 @12:54PM (#23435880) Homepage

    The metrics used in this paper are lame. They're things like "number of #define statements outside header files" and such.

    Modern code quality evaluation involves running code through something like Purify, which actually has some understanding of C and its bugs. There are many such tools. [wikipedia.org] This paper is way behind current analysis technology.

    • Re:Stupid metrics (Score:4, Insightful)

      by Diomidis Spinellis ( 661697 ) on Friday May 16, 2008 @01:15PM (#23436214) Homepage
      It took me about two months of work to collect these metrics. Yes, running in addition the code of the four kernels through a static analysis tool would have been even better, but this would have been considerably more work: You need to adjust each tool to the peculiarities of the code, add annotations in the code, weed out false positives, and then again you only get one aspect of quality, that related with bugs, like deadlocks and null pointer indirections.

      Using one of the tools you propose, you will still not obtain results regarding the analysability, changeability or readability of the code.

  • by mlwmohawk ( 801821 ) on Friday May 16, 2008 @12:59PM (#23435968)
    Sorry, I've been in the business for over 25 years and had to hear one pin head after another spout about code quality or productivity. Its all subjective at best.

    The worst looking piece of spaghetti code could have fewer bugs, be more efficient, and be easier to maintain than the most modular object oriented code.

    What is the "real" measure of quality or productivity? Is it LOC? No. Is it overall structure? no. Is it the number of "globals?" maybe not.

    The only real measure of code is the pure and simple darwinian test of survival. If it lasts and works, its good code. If it is constantly being rewritten or is tossed, it is bad code.

    I currently HATE (with a passion) the current interpretation of the bridge design pattern so popular these days. Yea, it means well, but it fails in implementation by making implementation harder and increasing the LOC benchmark. The core idea is correct, but it has been taken to absurd levels.

    I have code that is over 15 years old, almost untouched, and still being used in programs today. Is it pretty? Not always. Is it "object oriented" conceptually, yes, but not necessarily. Think the "fopen,"fread," file operations. Conceptually, the FILE pointer is an object, but it is a pure C convention.

    In summation:
    Code that works -- good.
    Code that does not -- bad.
    • by Llywelyn ( 531070 ) on Friday May 16, 2008 @01:19PM (#23436278) Homepage
      There is a company that, at the heart of their business, exists a 6000 line SQL statement that no one understands, no one can modify, and occasionally doesn't work without anyone knowing why but a restart of the program seems to take care of it.

      It has lasted that way for a very very long time.

      Is it good code simply as function of its survival and (sort of) working?

      I tend to think of good code like good engineering or good architecture. Surely you wouldn't define good architecture as "a building that remains standing," would you? The layout of the rooms, how well that space is used, how well it fits the needs of the users, how difficult it is to make modifications, etc all factor in to "good design" and have nothing to do with whether the building "works."

      I am not sure you can put a metric to it anymore than I could put a metric to measuring the quality of abstract expressionism or how well a circuit is laid out--there may be metrics to aid in the process, but in the end one can't necessarily assign a numerical rating to the final outcome for the purpose of rating.

      That doesn't mean that there isn't such a thing as good quality and bad quality code.
      • Re: (Score:3, Interesting)

        by mlwmohawk ( 801821 )
        Is it good code simply as function of its survival and (sort of) working?

        "sort" of working is not "working."

        exists a 6000 line SQL statement that no one understands

        This is "bad" code because it needs to be fixed and no one can do it.

        Surely you wouldn't define good architecture as "a building that remains standing,"

        I'm pretty sure that is one of the prime criterion for a good building.

        Your post ignores the "works" aspect of the rule. "Works" is subtly different than "functions." "Works" implies more than mer
    • by Diomidis Spinellis ( 661697 ) on Friday May 16, 2008 @01:28PM (#23436462) Homepage
      Coding to achieve some code quality metrics is dangerous, but so is saying that code that works is good. Let me give you two examples of code I've written long time ago, and that still survives on the web.

      This example [ioccc.org] is code that works and also has some nice quality attributes: 96% of the program lines (631 out of the 658) are comment text rendering the program readable and understandable. With the exception of the two include file names (needed for a warning-free compile) the program passes the standard Unix spell checker without any errors.

      This example [ioccc.org] is also code that works, and is quite compact for what it achieves.

      I don't consider any of the two examples quality code. And sprucing bad code with object orientation, design patterns, and a layered architecture will not magically increase its quality. On the other hand, you can often (but now always) recognize bad quality code by looking at figures one can obtain automatically. If the code is full of global variables, gotos, huge functions, copy-pasted elements, meaningless identifier names, and automatically generated template comments, you can be pretty sure that its quality is abysmal.

      • I don't consider any of the two examples quality code.

        A philosophical point, if I may. What is the purpose of code?

        I think the problem is that many engineers think that code should fit some form of aesthetic that is really irrelevant for the purpose. So far, the attributes that people use for judging software code quality and productivity have almost nothing to do with what the actual code does or how it works.

        Algorithms are hard to quantify unless you know about algorithms. Code "hardness," or the ability
        • With a liberal reading of "if it works" you're right. You can say that if the code is functional, reliable, usable, efficient, maintainable, and portable, then it is of high quality. But this is a circular definition, because this is how software quality is defined [spinellis.gr]. As somebody else posted earlier, the quest for quality can lead you to an endless motorcycle trip on America's back-roads [amazon.com].
          • Re: (Score:3, Interesting)

            by mlwmohawk ( 801821 )
            You can say that if the code is functional, reliable, usable, efficient, maintainable, and portable, then it is of high quality.

            Not to put too fine a point on it, but this is too much concern over stuff that does not always matter.

            I agree "functional" and "reliable" are absolutely important.

            "efficient?" Only if efficiency is required or of any concern. How efficient is efficient? It is a balance of efficiency against economy.

            "Maintainable?" Sure, most of the time, but not always. Sometimes we toss stuff on
    • The only real measure of code is the pure and simple darwinian test of survival. If it lasts and works, its good code. If it is constantly being rewritten or is tossed, it is bad code.

      For me, good code is really easy to modify and update and fix problems and add to. So people actually do these things, and the result is that good code gets rewritten and 'tossed' a lot.

      On the other hand, take things like ld (and friends) which iirc has code in "func(int) int param; { body }" style... which hasn't been used for what 20+ years? There are so many improvements I would like to make to it, but the code is too crusty to easily make them. I'd call that bad code... even though it 'works' just f

  • It's a well known fact that code will always resemble the institution that produced it, to some extent. To describe the Microsoft code as "poorly structured" is likely a bit out of touch.

    The absolutely best kernel code is generally extremely beautiful and descriptive when dealing with the system's abstracts (with nice, long descriptive names for each function) and then unbelievably hellish and ugly in the sections that deal with hardware. Kernels represent an intersection between the idealistic system code

  • By the time you get a car/plane that's accepted by the "market", the insides are going to be of a certain quality, no matter what the process used.

    Did I get that right? :)

    So now the question is how long did it take to get that level of quality. And maybe there is a difference in quality but the measurement used is not sensitive enough, or not as appropriate or his conclusion isn't quite correct - he measured a difference, just didn't show in his conclusion ;).
  • by rthille ( 8526 ) <web-slashdot@ran g a t .org> on Friday May 16, 2008 @01:39PM (#23436590) Homepage Journal

    Statements like this: "Indeed the longest header file (WRK's winerror.h) at 27,000 lines lumps together error messages from 30 different areas; most of which are not related to the Windows kernel." Allow me to feel smug in my anti-Windows bias :-)
    • Re: (Score:3, Insightful)

      Ten years ago I wrote an article [spinellis.gr] criticizing the Windows API. Most of what I wrote then, continues to be true today. Based on that external view of Windows, and the BSODs I regularly see, I was expecting to find in the kernel many worse things. The header file you mention is a clear manifest of an inappropriate design, and I suspect that at higher levels of system functionality (say OLE or the GDI) there will be more parts of similarly bad quality.
      • by rthille ( 8526 )

        Not sure how much work it would be to analyse yet another system, but I'd love to see how NetBSD fares in this comparison...
  • by Khopesh ( 112447 ) on Friday May 16, 2008 @02:21PM (#23437580) Homepage Journal

    Isn't NetBSD the system filled with academics who insist upon clean, manageable, and portable code above all other standards? Too bad the NetBSD kernel didn't get judged here, I suspect it would have taken the cake.

    I still recall this exhaustive report [bulk.fefe.de] comparing several kernels' performance back in 2003 in which NetBSD pretty much beat the pants off of everybody else (note the two updates with separate graphs). The initial poor performance was due to an old revision, and upon seeing that there were some places in which the newer revision wasn't so hot, the developers fixed them and in only two weeks, NetBSD beat out FreeBSD on every scalability test. Their pragmatism and insistence on quality code finally paid off.

    Ever since seeing those charts, I've been waiting for Debian/NetBSD [debian.org] to come out...

  • Weird logic (Score:3, Insightful)

    by BokLM ( 550487 ) <boklm@mars-attacks.org> on Friday May 16, 2008 @02:34PM (#23437816) Homepage Journal

    "An earlier study on the distribution of the maintainability index [4] across various FreeBSD modules [35,Figure 7.3] showed that their maintainability was evenly distributed, with few outliers at each end. This makes me believe that the WRK code examined can be treated as a representative subset of the complete Windows operating system kernel."
    How are FreeBSD and Windows related ? You conclude something about Windows source code based on things you saw in FreeBSD source code ?
    • What I'm saying is that when we're looking at maintainability of a large operating system (FreeBSD) there are few outliers. Therefore, one can make the case that in another similarly large operating system we can get a representative picture of its maintainability by looking at a subset of its code. My conclusion is related to the Law of large numbers [wikipedia.org], nothing deeper or more complex.

      Think of my argument as looking at the people living in China and seeing that there are no areas occupied by giants or dwa

  • by ThePhilips ( 752041 ) on Friday May 16, 2008 @04:14PM (#23439276) Homepage Journal

    The high marks of Solaris and WRK in preprocessing could also be attributed to programming discipline. The problems from the use of the preprocessor are well-known, but its allure is seductive. It is often tempting to use the preprocessor in order to create elaborate domain-specific programming constructs. It is also often easy to fix a portability problem by means of conditional compilation directives. However, both approaches can be problematic in the long run, and we can hypothesize that in an organization like Sun or Microsoft programmers are discouraged from relying on the preprocessor.

    That subjective conclusion is precise effect reading too much into the metrics.

    Sun or Microsoft programmers need to support 2 and 2 platforms respectively. (Sun: SPARC and AMD64; M$: IA32 and AMD64). All portability are of boolean complexity.

    But FreeBSD and Linux run on dozen of platforms. I do not know how it is in BSD land, but in Linux first and foremost requirement for platform support, is that it has no negative side-effects on other platforms. Consequently, for example, under Linux most (all? - all!!) locking is still implemented as macros: on uni-processor system with preemptive kernel feature disabled all in-kernel synchronization would miraculously (thanks to preprocessor) disappear from the whole code base. To make sure that on such platform, kernel would run as efficiently as possible - without any locking overhead, because all the locking is not needed anymore.

    And that's single example. There are many macros for special CPU features: depending on platform it would be nop or asm statement or function call. No way around using macros.

    I think one of the points the author needed to factor in, is portability of OS. Without that, most metrics are skewed too much.

    P.S. Actually, Linux affinity to macros is often (at least from words of kernel developers) stems from poor optimization of inlined functions in GCC. Many macros can be converted to functions - but that would damage overall level of performance. In many places significantly.

    • This is a very perceptive comment. It goes deeper: Linux and FreeBSD can be (and often are) configured by end-users. These can tailor the kernel in hundreds of ways. In FreeBSD 6.2 I measure more than 340 kernel options. These are mostly handled by the preprocessor.

      Thanks for pointing this out.

  • by nguy ( 1207026 ) on Friday May 16, 2008 @09:56PM (#23442510)
    Those statistics are useless; nobody has any idea how these measures translate into correctness, robustness, or performance.

E = MC ** 2 +- 3db

Working...