Software Code Quality Of Apache Analyzed

fruey writes "Following Reasoning's February analysis of the Linux TCP/IP stack (putting it ahead of many commercial implementations for it's low error density), they recently pitted Apache 2.1 source code against commercial web server offerings, although they don't say which. Apparently, Apache is close, but no cigar..."

A bit late, aren't we?

[Anonymous Coward]

http://www.neowin.net/comments.php?id=12345&catego ry=main [neowin.net]

--r

apache 2.1?

[fishynet]

2.1 is'nt even out yet! the latest is 2.0.46!

Defect?

[Jason_says]

Reasoning found 31 software defects in 58,944 lines of source code of the Apache http server V2.1 code.

so what are the calling a defect?

How do they get to look at closed source?

[3.5 stripes]

And don't most NDAs for when they do let you look forbid any competetive analysis?

Or am I just too far out of that line of work to know how these things work?

What kind of BS test is this?

[dtolton]

They are comparing a development version to an un-named commercial web server?

Why don't they compare it to apache 2.0.46 if they want a newer, but release product? I expect they did, but they didn't get the results they wanted.

This is a development version, it's an odd numbered release for crying out loud.

I wouldn't be suprised to see this is bankrolled by M$. Let's compare IIS in development to Apache 2.1, and then see what IIS bug density rate is.

Bah!!

Absolute crap

[degradas]

I can't think of any reason why should anybody trust this analysis until they publish the methods used. Anybody can say "Hey, I tested something using my proprietary method, and $foo has more bugs than $bar!". Unfortunately, such tests really don't say anything substantial about the quality of software. IMHO.

Interesting, with or without modules?

[hughk]

If anyone has an Apache 2.1 dist around, they say they checked 58,000 lines - does this seem reasonable? Is this with any of the modules such as PHP or Perl or is this raw????

I know that Apache has vulnerabilities but it should come better than IIS. You can't realisticly give a verdict on IIS without looking at the libraries called.

As for the rest, I can imagine some commercial products coming in better, but not many.

Does it matter?

[pubjames]

So?

There are errors and there are errors. There are error that don't matter a jot, and there are errors that are show-stoppers.

I've worked on banking software containing code that was written in assembly for PD11s and developed over decades. The most horrible spaggetti code you could ever imagine. Why did the banks keep using it? Because for any particular input it always gave the correct output.

Years of bug fixing had made the code horrible and probably full of errors if you were looking at it from a purely theoretical/software engineering viewpoint. But from an input/output point of view, it was faultless.

That's so weird ...

[SuperDuG]

I found just the opposite.

Important Tech City, CA, July 7th 2003
For Immediate Release
Sbj: Apache beats other webservers

Recently we had our staff (some guys kid) look over the source code of 3 major webserver packages available, in that code nearly 8 million lines of error were found, but surprisingly the damned things still worked?!

We placed a performance test (click a link and see if porn comes faster) with apached and 3 other commercial offerings. Apache seemed to knock them all of the water, boy will those other three companies be mad now.

While we cannot tell you what the other three offerings were (that might make this whole thing more believeable) we can tell you that we think they're popular.

Here's the results

Apache ------------------- 104
Com 1 --------32
Com 2 -----------45
Com 3 ---------------53

As you can see by the clear test results, apache wins in all tests.

Since when are unfounded results from a company that doesn't explain what the "32 defects" were, newsworthy. Don't act like these guys are worth my time, this is bullshit.

Re:BSD codestyle...

[Anonymous Coward]

have you seen the kernel code?

about half of it is "comments", that are really arguments/fights between Linus, Cox and Russell.
(no kidding, if you ever read the kernel mailinglist you should already know this).

Did they include comments in the test?

to be expected from Open Source

[Illserve]

By its very nature, Open source will tend to fix important bugs and leave unimportant ones unfixed, while standard QA processes associated with commercial software will tend to fix little UI issues during the release schedule before dealing with vulnerabilities.

So seems pretty clear to me that in Open source, the ratio of showstopper bugs to miscolored widget bugs will be much lower than for commercial software.

Re:Wait a second

[AftanGustur]

Has Apache 2.1 been released as a stable, non-developmental release?

According to the official site [apache.org].
The latest 2.* relase is "2.0.46 " and version 2.1 is nowhere to be seen ....

So the question is : Which version did they audit ??

That was not the conclusion: RTFA

[arrogance]

As others have stated, the article states that "the difference in defect density between the two was not significant." Meaning that defect density, especially with such a small differential, has little bearing on the overall quality of the software. We know nothing of the severity, impact, etc of the defects: they could all be cosmetic for all we know. This is probably nothing more than a marketing strategy by Reasoning: publish a study without any details on a hotly debated topic and see how many people check out their site. It'd be nice if they had a downloadable version of their software to test drive.

FxCop [gotdotnet.com] is an example of a "defect" or code analysis tool. While I have NO idea of Reasoning's methodology, I know that with FxCop (which is specifically for .NET code analysis), you have to set it up to filter out the majority of its rules or you'll get 3000 instances of "You didn't name this variable the way MS says you're supposed to." FxCop is extensible though. The point is, not a single poster on this page (unless they work for the companies involved) knows what Reasoning's methodology or rule set was when they did this so we can glean virtually zero value from this analysis. I look forward to 600 anti-Microsoft posts because of it though....

Re:Code defects appear to be a small part of the e

[AftanGustur]

Is IIS just inherinetly insucure because it is used on a Windows platform? Is it because hackers generally target IIS and not Apache (most people will rush to this conclusion)?

Microsoft will try to make people belive whatever is in their interests .. Even if it means contradicting themselves ..

Last Friday Microsoft called all their Premier customers in France with "information" related to the upcoming "hackerfest" last Sunday.

According to Microsoft mostly Unix and Linux servers would be the target of the hackers but it did not exclude IIS Web servers to come under attack.

The FUD coming from MS is absolutely unbeleavable..

Re:So if they found them...

[tomstdenis]

Agreed. Things like splint often report "warnings" on code that shouldn't be. For instance

int some_func(char *somebuf)
{
if (somebuf == NULL) return ERROR;
somebuf[0] = 'a';
return OK;
}

Will generate a warning with splint saying "pointer may be null" despite the fact it cannot be.

Those tools are generally too sensitive and give too many false positives to be useful in the long run.

Tom

Some "defects" aren't really...

[peerogue]

Look at defect ID #26 in the report.

You'll see that this can only happen when nItems is 0. This means that if a pre-condition was added to the routine tsort() that the nItems argument MUST be strictly positive, defect #26 vanishes.

If I'd put:

assert(nItems > 0);

at the routine entry, it would prevent the further null-pointer dereference and spot the bug immediately when it occurs. I'm not sure how well a web-server crashing would be perceived, but that would not be worse as a kernel panic'ing, and there is indeed a potential bug there.

My point is that to call #26 a defect (or not), we'd have to check all the callers, and if all the callers were to guarantee that nItems is strictly positive, then there would be no bug at all.

Apart from this remark, I think that kind of work is really great. I'd love to see it applied to my favorite open-source Linux Gnutella client (all Gnutella clients are by definition an HTTP client/server). We'd see how a small open-source project compares to a big one.

Re:Code defects appear to be a small part of the e

[AftanGustur]

Maybe that's because the majority of web servers are running on Unix/Linux?

True, but according to statistics [attrition.org] 56% of defaced webservers run Microsoft IIS, and (only) 34% Apache..

This is not brand new data, but it is the latest I can find ... And If Microsoft had some stats showing different results, you can be sure they would publish them..

The competition was about defacing 6000 webservers in 6 hours, so one would tend to conclude from the above that Microsoft IIS would be the primary targets..

Useless information presented confusingly

[albin]

Slashdot's summary of this article is way off base, and the article itself couldn't be less useful. Counting the number of "errors" in lines of code... and the ratio is supposed to mean something to us? As compared to unnamed other software? C'mon, I have better things to do with my time.

*plonk*

Re:Code defects appear to be a small part of the e

[bwt]

One of the best ways to get to know a large code base like Apache or something else is to find a repeatable bug and track it down. To fix a bug you do not need to understand the whole program, just the relevent parts. I've submitted bug fixes to several projects, so I must strenuously disagree, especially because, ahem, I have never submitted a bug fix to a proprietary project because its impossible.

Re:Recursion

[nick255]

Yes the proof is quite a simple application of the famous halting problem proof.

Imagine you made the program go into an infinite loop whenever the program it was analysing did not have an infinite loop.

Them run the program on itself......

Apache 1.3?

[Spazmania]

First, as many posters have noted, Reasoning DID NOT TEST APACHE 2.1. They tested Apache 2.1-dev. That's dev, as in development branch. As in: I have new untested code, so don't use me on a production server until I'm released in the STABLE series.

For a valid comparison versus commercial software, the testers should have used Apache 2.0.46, the most current STABLE series release.

Second, I'd be interested to see a comparison of 2.0.46 versus 1.3.27. I have a pet theory that multithreaded C code has more bugs than single-threaded C code, and I'd like to see whether there is evidence to support it.

Re:Code defects appear to be a small part of the e

[jdh-22]

You have the wrong idea here. There is a point in which you must realize what information you can release without comprimising the security of your system. While I can give you the plans to my vault, I will not give you the combination, nor the first or second numbers in it.

For the star wars geeks out there, if you were a Jedi, you don't go around telling everyone you're a Jedi, nor do you flash your light saber in public places. They do realize when to show their light saber, and when they can tell people they are a Jedi. Nor do they not tell anyone who they are, or never show their lightsaber.

You might want to check out Secrets and Lies [amazon.com] which will give you a better understanding of security philosphy.

Re:Defect is too strong a word...

[DrInequality]

Defect is way too strong. Take Defect 1. Can only possibly derefence a NULL pointer if a number of preconditions are true. The last one is (!conf->providers)[the pointer in question] must be false.

!!conf->providers => conf->providers => conf->providers != NULL

Their program has detected "defects" where there are none. Perhaps the greater coding style variation on open source projects exposes more defects in their automated program!

Re:Code defects appear to be a small part of the e

[aziraphale]

One word: architecture.

And not just the architecture of the web server, but the architecture of the entire platform. But specifically looking at the architecture of Apache versus the architecture of IIS, you'll immediately see that the goals of the two pieces of software are not the same. Look at things like IIS's metabase - the structural details of the server's configuration are kept in an in-memory data structure, which is easily modified while the server is running. Apache, in contrast, reads its configuration at startup, and uses it to determine which modules of code are loaded, and how they are used to process requests - fixing the behavior of the web server at startup.

IIS follows typical MS enterprise software design - it has to interface with COM, and the NT security model, and active directory, and the registry, and a million other systems, all in the name of integration, and enterprise management. Apache doesn't have PHBs telling it that it needs another way for the metabase to be edited, or a new instrumentation API, or whatever else a particular large customer asked for - and can get on with just providing its facilities cleanly.

That's why IIS has so many more security holes, even if it does (as may or may not be the case) have the same raw coding error rate as Apache.

Re:So if they found them...

[Error27]

FUD??? Gimme a break.

It says pretty clearly that they purposely chose a less mature sample of open source software than they did last time. The point is, does open source software start out bug free or do the bugs get worked out with age?

Worst kind of science

[AYEq]

Reasoning's code inspection service is based on a combination of proprietary technology and repeatable process.

Am I the only one who looks at reasoning's results with suspicion (even when I agree with them). Any analysis using methods that are not open and repeatable is not science. This just feels like marketing to me. (it is sad because the study of code quality is such a worthwhile pursuit)

Re:Recursion

[Anonymous Coward]

Why couldn't it ever work? Alls you have to do is build a table of all the variables the loop condition depends on (and the variables they depend on, and so on). Then as you execute the loop, you put the values of each of the dependent variables into the table. You procede until either the loop ends (in which case it's obviously noninfinite) or until you detect that all the dependent variables in one loop instance are exactly the same as the depedent variables from a previous loop instance. Obviously, this is depending on the fact you are using fixed size numbers which gaurentees that an infinite loop will traverse over at least one set of depedent variables more than once. But seeing how computers are finite machines anyway.....

What happens if you get something like this:

int k=0;
int i=0;
while( i < 20 || k < 365 )
{
k++;
}

As in you forgot to put

i++;

in the loop.

The snapshot of every loop's dependent variables is different, but it's still an infinite loop because 'i' never increases. Keep in mind this is just a counter example, and of course you could modify your idea to make it work in this case. However, somebody has formally proven that you can't make an infinite loop detector. IIRC, the book "Godel, Escher, Bach" has some interesting stuff on this and other issues with AI.

prove it.

[Mark19960]

they dont say what they used for a comparison.
when they tell us what they used, then I will believe it.
this smells microsoft.

bring it on! we want to know what it was compared against, sure as hell was NOT IIS...