Performance Benchmarks of Nine Languages 954
ikewillis writes "OSnews compares the relative performance of nine languages and variants on Windows: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, Python compiled with Psyco 1.1.1, Visual Basic, Visual C#, Visual C++, and Visual J#. His conclusion was that Visual C++ was the winner, but in most of the benchmarks Java 1.4 performed on par with native code, even surpassing gcc 3.3.1's performance. I conducted my own tests pitting Java 1.4 against gcc 3.3 and icc 8.0 using his benchmark code, and found Java to perform significantly worse than C on Linux/Athlon."
Re:Trig functions... (Score:4, Informative)
For many math functions java uses a software implementation rather than using the built in hardware functions on the processer. This is to ensure that these function perform exactly the same on different architectures. This probably accounts for the difference in performance.
heres the link to the code (Score:2, Informative)
Alternative comparison, compiler shootout (Score:5, Informative)
Re:Sitting on a Benchmark (Score:4, Informative)
Ximian's Mono has a C# compiler for open OS's:
http://www.go-mono.com/c-sharp.html
Re:Trig functions... (Score:3, Informative)
Re:Trig functions... (Score:2, Informative)
I, for one, would _never_ trust Java in a mission critical embedded environment. In fact you still see assembly in those envrionments from time to time. Imagine using Java for a fly by wire system. Would you fly on a plane that was using Java for fly by wire? I, for one, would not.
Java is great for some things. But you get too many cases where companies use a new technology without adequate due diligence simply because its the NTOW (New Technology Of the Week). I still say that a server written in C (written properly of course) will outperform a server written in Java.
Re:Why did VB do so bad on IO. (Score:5, Informative)
Re:Trig functions... (Score:5, Informative)
Quoting the results section here... (Score:5, Informative)
Results
Here are the benchmark results presented in both table and graph form. The Python and Python/Psyco results are excluded from the graph since the large numbers throw off the graph's scale and render the other results illegible. All scores are given in seconds; lower is better.
int long double trig I/O TOTAL
Visual C++ 9.6 18.8 6.4 3.5 10.5 48.8
Visual C# 9.7 23.9 17.7 4.1 9.9 65.3
gcc C 9.8 28.8 9.5 14.9 10.0 73.0
Visual Basic 9.8 23.7 17.7 4.1 30.7 85.9
Visual J# 9.6 23.9 17.5 4.2 35.1 90.4
Java 1.3.1 14.5 29.6 19.0 22.1 12.3 97.6
Java 1.4.2 9.3 20.2 6.5 57.1 10.1 103.1
Python/Psyco 29.7 615.4 100.4 13.1 10.5 769.1
Python 322.4 891.9 405.7 47.1 11.9 1679.0
Re:Wow (Score:2, Informative)
Dan
Re:Under Windows... (Score:3, Informative)
Re:Wow (Score:2, Informative)
GCC, like apache, is meant to be correct and portable first, fast second. Despite this, I wouldn't say thats its performance sucks, I would say that it is the fastest cross-platform option available. (as compared to Java the only other cross platform non-interpreted language in the test group).
Python Longs are arbitrary precision! (Score:4, Informative)
It is an arbitrary precision decimal type! That's why Python's scores on the Long test are so much higher (slower) than the other languages.
I wonder what Java scores when the benchmark is reimplemented using BigDecimal instead of the 'long' machine type.
Python uses a highly efficient Karatsuba multiplication algorithm for its longs (although that only starts to kick in with very big numbers).
Re:Trig functions... (Score:3, Informative)
exec $JAVA_HOME/bin/java.bin -Dsun.java2d.pmoffscreen=false $*
That way any plugins or Web Start apps also get this treatment. Just make sure you also change the JRE copy of java, too.
Re:Under Windows... (Score:1, Informative)
trig calls in gcc (Score:5, Informative)
You can enable inline trig functions in gcc as well, either with a command line flag, or an include file, or by using "asm" statements on a case-by-case basis. Check the documentation. With those enabled, gcc keeps up well with other compilers on trig functions.
Re:Under Windows... (Score:2, Informative)
Python waits for input just as fast as all the other languages.
Development and benchmarking of 1 million human typed lines is left as an exercise for the reader.
Oh, and my parsing recognizes verbs, nouns, objects and pronouns. All of this is easily optimized to run fast enough in Python. I'm sure it could eventually be made faster in C but I won't even explore that.
Re:Trig functions... (Score:4, Informative)
Isn't that what AWT tried to do originally? I'm just delving into Java for the first time the last few months, but I thought I've read this in "Core Java, Vol. 1"
They say (pg. 236 "Core Java, Vol. 1) that this resulted in a "write once, debug everywhere" problem since you will have different behavior, different limitations and different bugs on each implementation of AWT on each platform
Re:Why did VB do so bad on IO. (Score:5, Informative)
Re:Trig functions... (Score:3, Informative)
By calling Swing a crappy library you show that you really haven't used it all that much if at all and are just puking tired, second-hand information.
Re:Python Longs are arbitrary precision! (Score:2, Informative)
Regards,
Steve
Re:Why did VB do so bad on IO. (Score:2, Informative)
C#
StreamWriter streamWriter = new StreamWriter(fileName);
while (i++ ioMax)
{
streamWriter.WriteLine(textLine);
}
streamWrite
VB.NET
FileOpen(1, fileName, Microsoft.VisualBasic.OpenMode.Output)
Do While (i ioMax)
PrintLine(1, myString)
i += 1
Loop
FileClose(1)
Totally different routines, totally different IL, totally different performance. If he'd had simply stuck to the
Re:Trig functions... (Score:3, Informative)
Conversely, when you have a finite number, 1-thread-per-stream asynchronous IO is VERY desirable; both in coding, and in some small ways even in efficiency (immeidate responses; limited only by context switching time).. In contrast, IO-multiplexing may have certain tasks take long periods of time (stalling other channels).. Then you have to manually keep track of such situations as you code and push them off to worker threads in certain types of problems.
But there is simply no substitute when you want thousands of simultaneous idle connections (such as message servers) for IO-MUX.
Not that it's better, just solves a particular solution.
Tester does not understand Java (Score:3, Informative)
I dont know why the reasons are not clear to him. Perhaps its because he still thinks the JVM is "running bytecode" and does not understand what JITs did or what HotSpot compilers do. Byte code is only run the first few passes, after which its optimized into native code. Native being whatever the compiler of the c program used to compile the JVM could do. This is fundamental. Which explains his results, and points to a poor HotSpot implementation where trig functions are concerned.
Re:Trig functions... (Score:4, Informative)
Re:Trig functions... (Score:2, Informative)
The factors:
1 - fast-enough machines (response time 1/10 second is not noticable - no one cares if that is 10x slower than native at that point. you can't see it.)
2 - Improvements to Swing. 1.4 is the first version of Swing which has hardware acceleration built-in, thereby putting it on par with native win apps (native apps would be terribly slow without HWaccel, too - just turn it off in your graphics card settings to see how)
3 - and, to a certain extent, enough memory in machines. Java is a memory hog (even a small client app uses 20M...). But that doesn't matter if your machine has 256M.
You can say Swing is still overkill and over-engineered, and the AWT is no beauty, either. All true. Only, at this point, it doesn't matter. The "perceived suggishness"-barrier has been passed and left behind for good.
If your Java GUI is sluggish in 2004, you have only yourself to blame.
Ongoing, open source "language shootout" (Score:3, Informative)
http://scutigena.sourceforge.net/ [sourceforge.net]
It's designed as a framework that ought to run cross-platform, so you can run it yourself. We haven't added it yet, but I think we really want to divide the tests into two categories. "Get it done" - and each language implements it the best way for that language, and "Basic features comparison" - where each language has to show off features like lists, hash tables, how fast function calls are, and so forth.
It's an ongoing project, so new participants are welcome! I would appreciate it if comments went to the appropriate SF mailing lists instead of here, so that I can better keep track of them.
Re:Trig functions... (Score:5, Informative)
Re:Trig functions... (Score:5, Informative)
I'm only a beginner in C# and Java, but I know both have reflection, and the proposed Java 1.5 has enums. Kudos to C# for having them first
Also
Better for whom? Why? Doesn't it have the severe shortcoming of platform lockdown?
I can write a c#.net app in 1/4th the code of a java one. Go take a look at Microsoft's petshop program if you do not believe me.
I can write an assembly app in 1/4 the code of a Python one. Assuming, of course, that the Python app wasn't written for small code size... The simile is very accurate; Sun didn't write their petshop for small size.
The Java Petshop reimplementation here [prevayler.org] spanks both Sun's and Microsoft's petshop in terms of size, and pretty clearly demonstrates that both languages could do better.
BTW, I absolutely love C# -- from what I've done with it so far. My only complaint is that its support is at best halfhearted for other platforms, and I will not allow my work to be tied down to one platform. This is the only thing that kept me from learning K (well, K is portable, the only problem is that it's only available from one vendor, Kx systems). Anyhow, I think C#'s bytecode is far beyond anything Sun's ever going to do with Java.
ALso WIndows2k3 is as stable as Linux now. NT4 is old. The situation has improved dramatically. I have never even seen a blue screen on windows2k yet!
I agree with all of that, but it's not enough. I have seen blue screens and system crashes on 2000 and XP (XP far, FAR FAR more often than 2000). But then I've seen system crashes on Linux, so I'm not just complaining about MS
-Billy
Re:Trig functions... (Score:2, Informative)
AWT/SWT only uses low level calls (probably as low as : draw this pixel at x,y with this color) and recreates all the widgets "from scratch". And if you ever have the time, go download the source code of java on sun's website (there is a community edition to download for free ) and take a look at the implementation of swings components, no wonder they are sluggish. I won't even get started on the focus management of swing which is "weird" to say the least.
Re:Java Performing worse then C (Score:3, Informative)
The problem there is if you start taking a truly open look around, Java stops looking so cool either.
Its single greatest feature that might justifiably be called unique is the fact that it has a lot of library support you can buy, and if those things help enough they might tip the balance.
But in every other way, there's a language that meets or beats Java on almost every level.
To prevent this from becoming too confrontational, I won't try to name the languages. But Java really isn't the king the pro-Java zealots like to pretend it is; it's just the current Big Kahuna in certain domains, just like C++.
Tried this with gcj 3.2, here are the results (Score:4, Informative)
Comparison against gcc, gcj and Java 1.4.1 on the same host:I was somwhat surprised on the difference in the trig tests, as both appear to use libm. Not surprised that the IO was slower, the Java IO classes are nifty but do add quite a bit of overhead compared fputs/fgets.
(Sorry about the formatting, it was the best I could do)
Python Benchmark (Score:4, Informative)
$ python -O Benchmark.py
Int arithmetic elapsed time: 13700.0 ms with
Trig elapsed time: 8160.0 ms
$ java Benchmark
Int arithmetic elapsed time: 13775 ms
$ java -server Benchmark
Int arithmetic elapsed time: 9807 ms
(n.b. this is only a small subset of the tests- I didn't feel like waiting. Trig was not run for java because it took forever.)
To dismiss a few common myths...
1) Python IS compiled to bytecode on it's first run. The bytecode is stored on the filesystem in $PROGNAME.pyc.
2) the -O flag enables runtime optimization, not just faster loading time. On average you get a 10-20% speed boost.
3) Python is a string and list manipulation language, not a math language. It does so significantly faster than your average C coder could do so, with a hell of a lot less effort.
Re:About the Python performance (Score:2, Informative)
Re:Trig functions... (Score:4, Informative)
AWT has native widgets: Combo box, menu, button, text area, input box, checkbox, etc... Not only primitives.
What you are describing is Swing, not AWT.
Swing relies on the most basic AWT features: Component/Container and drawImage, and re-implement the whole widget sets in Java, relying on these two AWT components.
Re:Benchmarks? Who cares? (Score:3, Informative)
Not only that, but it helps to have a benchmark that actually tests the things that are claimed at being tested.
It's a Good Thing (tm) this didn't make the front page.
The author states "I am by no means an expert in benchmarking; I launched this project largely as a learning experience" and it shows. The man has an associates degree in computer science, and a Ph.D. in psychology. His list of publications [berkeley.edu] are his dissertation, a single published paper, and excerpts within a 15-year-old travel guide.
Just a cursory glance at the first page of his article shows that he has no clue as to how things work. He states "I first tried to eliminate the CLR from the Visual C++ benchmark by turning off the language's "managed" features with the #pragma unmanaged directive, but I was surprised to see that this didn't lead to any performance gains." If he cannot understand why something that generates a few thousand CPU instructions of initial overhead doesn't change the speed of an I/O- and loop limited program, he isn't skilled enough to interpret his own results.
His benchmarks never actually test his first, second, third, or fifth question. His fourth question is actually addressed better when his contrived test is compared on relative measure with his two Java tests.
Look at his benchmark programs (found here [berkeley.edu]). Some of those test can, should, and will have compiler-specific optimizations, having nothing to do with the language. General 'counting loops', which is the only thing he is using, have long been known to produce bad benchmarks. He claims to be testing 64-bit floating point math, but in fact, many of his examples use 80-bit floating point.
Just for fun, look at his VC and Java 1.4 floating point tests. Now look at his compiler options. It is painfully obvious that the compiler saw "He explicitly said this is a Pentium 4, I can use parallel floating point instructions!" where the other compilers could not. Saying that those languages are inherently faster than the other compiled languages is lunacy.
This is hardly news. This is a BAD example of benchmarking, and would be given a poor grade in a graduate level CS class.
Microsoft's trig "functions" aren't (Score:3, Informative)
That would explain a lot then.
Any recent version of Visual C++ supports "intrinsics", i.e., conversions of a function call directly into a specific machine code instruction, or "perfect in-lining" if you prefer.
That means that if you use something like abs(x) in C++, these compilers would convert directly into an instruction to get x to the head of the floating point stack, then a FPU abs opcode, with no overheads at all. Same goes for trig functions and other variations supported by the FPU.
Moreover, IME the VC++ optimiser is quite smart about intrinsics, and in lengthy calculations will often arrange for the right values to be coming to the head of the stack in the correct order as cheaply as possible, even if it means planning ahead a few instructions. If you look at the assembly language output from VC++ for a numerical computation, it tends to have a series of instructions to stack what it needs, with the occasional calculation opcode thrown in between them, and then a whole series of neatly co-ordinated calculation opcodes at the end.
I don't see how any language that has function calling at all could keep up with this low-level, direct-to-FPU approach, and surely any language using software emulation of floating point code won't even come close.
I don't know whether VC# and VB.Net use the same trick, but given how much effort MS put into sharing things across the product range prior to releasing VS.Net, it must be a good bet. There are some advantages to writing platform-specific code. :-)
.NET is not Windows only (Score:2, Informative)
Re:Trig functions... (Score:4, Informative)
Or just doesn't bother implementing it at all. Try printing from eclipse on Linux.
Wha? (Score:3, Informative)
In other words, idiotc.
Re:Trig functions... (Score:1, Informative)
Trigonometric functions are not part of the Java language but they are the Java library.
I think it is slow for the trig functions because of the overhead to call the native OS trig function (switch from Java to native).
They are probably implemented something as this in class Math:
public native double cos(double v);
and the native funcion is:
jdouble Java_lang_Math_cos(jdouble v) {
return cos(v);
}
The switch from Java to native code involves some overhead. For I/O it does not matter much, I/O is already slow. But if you have trig function in a tight loop, then you have quite some overhead unless the JIT is smart enough to handle special cases and inline directly the cos(v) native code.
Re:Trig functions... (Score:3, Informative)
Its actually a pretty easy optimization: if a reference to a an object doesn't escape the current scope (which they can determine they usually analyze all or most of a program at the same time) then it can stack allocate the object.
Re:Trig functions... (Score:2, Informative)
sorry to say, but this is ignorant. eclipse is a great app and netbeans might have a sluggish gui, but i would not ditch an application because of its toolkit...
a great example for a good swing-app is intellij idea [intellij.com] and its a hell of an IDE (IMHO the _very_ best java-IDE _by far_) use it one day and you never want to go back to any other java IDE. its pure swing, looks good, feels good and is fast and responsive!
yes, its commercial, but thats not the point in this discussion (but you can get a trial-licence, so i'd suggest to try it)
Re:Time to defend C++ (again)... (Score:3, Informative)
I agree with 2/3 of your replies, but you need to look up the difference between C++ destructors and Java's finalize(). Hint: one is useful, the other usually isn't. :-)