Another J2EE vs .NET Performance Comparison

Starting yesterday, we received a bunch of story submissions about a performance comparison between J2EE and .Net. It didn't seem all that exciting, and we sort of ignored the story. But as usual, it appears that some people take issue with the methodology and conclusions.

J2EE

[Anonymous Coward]

We used all java technology on www.bayoubid.com [bayoubid.com] and had no problems with speed. In fact from our initial tests java was quicker than C#.

Paid for by MS

[Anonymous Coward]

Several independent sources have now confirmed that The Middleware Company was indeed paid by Microsoft to conduct this report.

From second article.

Much more interesting - a leaner JPetStore

[slamb]

jPetStore [ibatis.com] is worth checking out. These people decided that the J2EE pet store is way too complex, which I'm inclined to agree with. They produced, using Jakarta Struts, a Java pet store web application that is much leaner. It's comparable in size to the .NET pet store but better in several ways - there's no SQL embedded in the code, there's no HTML embedded in the database, no code was automatically generated, and it's MVC-based.

I've always thought that Enterprise Java Beans are overengineering to the extreme. It's nice to have something to back that up with now. There's no question in my mind that this JPetStore beats out both the original J2EE one and the .NET one in maintainability.

They didn't do any benchmarks - performance wasn't what they were going for - but it would be interesting to see some. I'd be inclined to believe this simpler approach would also have much better performance than J2EE.

Re:Paid for by MS

[Call Me Black Cloud]

Not exactly. TMC explains it in their FAQ [middleware-company.com].

Re:Save your time

[interiot]

Actually, it's the other way around. Look at the PDF [middleware-company.com], page 40.

• The Middleware Java Pet Store 2.0 implementation uses the same basic EJB-recommended architecture as the original Java Pet Store (except fully optimized for performance). Hence, its code count remains largely unchanged over the original at 14,004 lines of total code.

• With the .NET Pet Shop 2.0 implementation, Microsoft has done some further optimizations to reduce its overall line count, while also extending the application with new features for distributed transactions and Web Services. The new .NET Pet Shop 2.0 contains a total of 2,096 lines of C# code (the 1.5 version had a total of 3,484 lines of code, a 40% reduction).

This is covered right away in the rebuttal, as there seem to be some tricks played to get the discrepancy so large.

Re:Paid for by MS

[ruiner13]

I noticed that too. Funny thing is, being a Java developer myself, I looked at those benchmarks and came to the same conclusion. We run J2EE servlets and beans on a 350MHz machine and we have not had any performance problems. I smelled something fishy. Then I read the second article and busted out laughing when it said:

"Several independent sources have now confirmed that The Middleware Company was indeed paid by Microsoft to conduct this report."

My instant reaction was... "Well DUH!" At least they didn't make up someone who used that as their reason for switching...

One JavaBot's Reply

[FortKnox]

From the Benchmark FAQ [middleware-company.com]:
No Local Interfaces were used, so it wasn't a truely 'apples-to-apples' comparison. Local Interfaces make a huge difference (although, they claim to of not seen one in another container).
They used the latest .NET technology, but didn't employ EJB2.0. But, what's worse, is that they used Entity beans. .NET used a stateless model, but they didn't do the same thing with the Java app.

I'm not going to tell you J2EE is faster, but I am going to say that they did a lot of apples-to-oranges comparisons and not note them (except in the faq).

Re:Save your time

[harvardian]

Lines of code has a little bit to do with reliability. It's a well-known fact that the more lines of code you write (called SLOCs), the more bugs you will have (notes on this here [iastate.edu]). Although more SLOCs means more bugs, density of bugs does not increase with code length (IEEE report here [computer.org]).

Re:Save your time

[Gaijin42]

There weren't tricks to get the descrepancy large. Sun wrote the original J2EE version. MS wrote the .net version. If Sun chose to write their code in an inefficient manner, (and you care about lines of code) then perhaps that tells you something about OTHER apps that sun writes?

Re:Infringement....

[pbranes]

If you were to read the article, you would find this text at the bottom, "Update 2002-Oct-30 Several independent sources have now confirmed that The Middleware Company was indeed paid by Microsoft to conduct this report." Therefore, they obviously had approval from Microsoft to conduct the benchmark.

Re:One JavaBot's Reply

[JonK]

And, as they say in the FAQ, one of their app servers didn't support EJB 2.0, so all the wailing and gnashing of teeth and rending of clothing from the JavaBots is largely irrelevant - if it don't run in the real world, it doesn't matter how great it would theoretically be

. And, as the FAQ you've so kindly linked to points out, any competent app server will make cross-component calls between components in the same process as in-process calls - thereby negating the main marketectural advantage of local interfaces. Like TMC say, they've done the most apples-to-apples comparison they can, but, again as they point out, there are always going to be problems where the zealots can point out the differences.

But hey, I'm a C++Bot so I can play disinterested observer. But I will tell you that C++ would be faster, safe in the knowledge that no-one's going to be going to the effort to put that benchmark together ;-)

was MS involved?

[loconet]

From their FAQ [middleware-company.com] about the benchmark..

Was Microsoft involved in this, did they fund this, where were the tests done?

Yes, Microsoft was certainly involved, as the paper describes. The Middleware Company approached Microsoft regarding performing such an experiment. Microsoft provided the lab, which was located in Seattle, funded the setup costs, and reimbursed us for expenses, including travel expenses. The Middleware Company invested several man-months in this project for which it received no compensation. The activity took much long than we expected, and at various points, we also hired independent consultants experienced in appservers A and B to tune them or provide recommendations, at our own expense. The parameters of the lab were under the control of TMC. TMC controlled the testing process. TMC stated up front that TMC would write a report about the real results, no matter what they were. These experiments are time-consuming, and require resources. Without permission and some support from Microsoft, we would not have been able to conduct the experiment. We would like to have conducted many more experiments than we did, and hope to in the future. TMC stands behind the results of the tests that were conducted.

Does the fact that Microsoft gave permission for this experiment and provided some support, invalidate the results?

That is for you to decide. TMC stands behind the results of the tests that were conducted. Should there be other such experiments to be arranged in the future, we will not be able to do them without some assistance with the lab, setup, expenses, and we would hope for more support than Microsoft provided us with for this experiment.

Re:Some of us

[Twirlip of the Mists]

Yes, the article does clearly state that TMC should be called to task... but then it fails to give any good reason for it. Every criticism of TMC's methodology is a debatable one. They point out, for example, that the LOC comparison is unfair, but then they speak only in vague terms about "excessive exception handling" to explain why.

Why don't you read the article, and then why don't you fuck off.

Re:I have no problem with microsoft's coders..

[plumby]

for example, a bean represents a row, not a business object, unless your business object is a row, in which case you are probably over exposed to changes in the database

I'm assuming that you are referring to Entity EJBs, as a 'bean' is basically any java object with a null constructor (and usually setter and getter methods), and can represent anything you want. But even for Entity EJBs, it's only true if you use CMP (container managed persistence). If you implement BMP (bean managed persistence) you can use them to represent any kind of business entity. We are currently implementing an entity bean that represents data that has been pulled off a mainframe (not an RDBMS in sight), with the EJB/container providing the cacheing/security/pooling etc for us.

Re:Here are the reasons why this comparsion is BS

[djaxl]

Couldn't find the 10 times faster than .Net comparison by Oracle, but:

http://www2.theserverside.com/resources/article.js p?l=PetStore [theserverside.com]

http://www.oracle.com/features/9i/index.html?9iasf ast.html [oracle.com]

The latter gives a measurement of 538ms response time for 600 users on the Petshop. Since the Middleware Company PDF lists response time as 14ms for "J2EE AppServer A" and 10ms for .Net, at 750 users, there's obviously some difference in benchmarking practices. If anyone has a link to the "10 times faster" propaganda, please post.

Did no one read the other article linked to?

[j3110]

I've been using J2EE now for a while, and they made some hideous performance mistakes. The #1 mistake they made was BMP. BMP, for those of you who don't know, is an object persistance model where each object manages it's own storage. It's pretty obvious that for N rows of a database that map to N objects, you will need N SQL statements. That's just wrong and bad. Not only is it the slowest way to access the database, but it requires 10x the amount of code to work with. The other two (common) choices are CMP and JDBC in session beans (others are JDO and other ADO-ish Java API's that wrap JDBC). CMP would require 1 SQL statement to retrieve N rows, and requires no SQL be written, works on all RDBMS's, and you only have to write a skeleton object. It's about a magnitude of 3 faster than CMP. Directly connecting from the Session beans(pretty much a CORPBA object) will make you write your own SQL, but will increase performance yet further(since you can use stored procedures or just do mass updates and still maintain database independance).

The next thing they did is exclude JBoss, one of the most popular J2EE servers. It's open source, and easy to use. One can only conclude the intentionally left it out because .Net could never hold up to best price/perf. against free. JBoss may not be a speed deamon(it's not slow at all though), but if you disable debugging(on by default), use IBM JDK 1.3.0 and MySQL with innoDB, it will easily win price/performance.

After reading that TMC had taken money from MS, the only conclusion that I could come up with is that it was rigged. No real J2EE expert would ever make those mistakes. Even free E-Books on TSS will tell you not to make mistakes like that.

Basically, this really hurts the Java community to see TMC take stabs at J2EE after all it's put into it. Either that or we are to conclude that TMC is unfit for the J2EE educational services they offer. Either way, they may have helped .Net get a foothold, but they are loosing theirs fast.

Anyone that doesn't know that much about J2EE or doesn't take a look at the code will think this is like the florida recount fiasco, but it really is a legitamit claim that this version of the petstore was really written by A) a monkey, or B) a MS fundee.

Re:Some of us

[malfunct]

You can use a simple text editor to edit .NET software and compile with a command line compiler (distributed with the .NET SDK).

Re:Save your time

[sfe_software]

Now of course the 14000 lines in the .Net application...

I'm extremely curious, as many people have mis-quoted this figure. Where did you get this information? Is there another article that quotes this incorrectly?

The .NET solution only had 2096 lines, while the J2EE one had 14,000+ lines of code...

So much for Microsoft's write tons of shit code and hope for the best mentality :p

Re:Performance isn't most important

[Fugly]

Only the fastest Java desktop applications are usable on my PIII 1.2GHz, namely NetBeans and Eclipse, and that's because they don't use Swing. I wrote a Hello World app in C# and it took 2 seconds to start. Language performance will continue to count until we all have 3+ GHz machines.

That's funny because my laptop is a PIII 1.2GHz and I use JBuilder 6 (considered to be one of the slower java IDE's) every day for java development. I thought it was usable but I guess all the applications I've written with it don't actually exist. Damn, I guess the last year of my life has been a figment of my imagination...

I'm not arguing that java and swing perform as well as native code in GUI environments. However, they've come a long way and they are clearly "usable". I'd even venture to say that the performance is "acceptable". Coupled with the fact that the applications can run on multiple operating systems without so much as recompiling the source, I might even call it "useful".

Re:Performance isn't most important

[dnoyeb]

Point of clarification for the ignorant.

J2EE is not a language. .NET is not a language.

Objective C is irrelevant in this comparison.