Van Rossum: Python Not Too Slow 510
snydeq writes "Python creator Guido van Rossum discusses the prospects and criticisms of Python, noting that critics of Python performance should supplement with C/C++ rather than re-engineering Python apps into a faster language. 'At some point, you end up with one little piece of your system, as a whole, where you end up spending all your time. If you write that just as a sort of simple-minded Python loop, at some point you will see that that is the bottleneck in your system. It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language, because for most of what you're doing, the speed of the language is irrelevant.'"
Re:007087 (Score:5, Interesting)
I have been coding in C for quite a while. Everything from simple user programs through kernel code, drivers, and firmware. I find that python is quite liberating. It allows me to accomplish a lot more faster and have it tend to just work. I do perform the analysis and occasionally re-implement a performance critical function in C, but I find that it's surprising how little that actually improves most situations, not because of a deficiency in Python, but because it wasn't actually all that slow.
Yes, I could implement the cool handling of various data types Python has in C, but doing so will produce something that looks a LOT like a poorly tested and less optimum implementation of Python.
Re:007087 (Score:5, Interesting)
Because it's significantly faster in terms of development time, or in terms of cost (having the less skilled, cheaper programmers work on it).
It's always been a case of write the critical stuff in C/C++. That's why languages like python and perl make doing so so simple.
That is heavily project dependent. If you are in a position where you can work with pure Python (i.e. the tedious job of writing wrappers for external libraries is done by others) then yes, you get RAD benefits but that is not chiseled in stone. If you are forced to combine your own C++ components with Python the picture changes. I am involved in a project that has a specialized OLAP engine, written in C++ for speed (no way around it) and a 3D GUI app that runs on top of it which was written in Python for RAD benefits. The biggest single headache we encountered with Python was that yes, you do code the GUI faster in Python, but you loose a lot of development time screwing around with writing Python wrappers for the C++ APIs of the OLAP engine. Plus, there are also endless deployment headaches. It's not as if every customer's machine comes preloaded with Python and even if it does, believe it or not your app may work with one Python distro but it will crash with the another and there are also incompatibilities between versions of Pyhon and you have no way of knowing what version is default on the target machine. You may end up having to bundle your own Python environment with your app for stability. Eventually the GUI team concluded that they were better off time wise if the just wrote the GUI in C++ using QT.
Re:007087 (Score:4, Interesting)
Amen to this:) The biggest issue with scripted languages seems to be that people think simplicity of the language is a substitute for proper design for the solution to your problem. I've been involved in coding a feature where there is a possible solution in both C++ and python; seems that C++ is seen as cumbersome mostly because it requires more forethought and python is then immediately hailed as the victor as soon as it can do a halfassed job of it.
Add to this that most people really can't code C++ and do not understand stl...
Re:It's still too slow, despite what he says. (Score:5, Interesting)
Says the guy whose whole life is tied up in the language
That's fair.
and whose project, at Google, to speed it up, crashed and burned.
That's completely wrong. Unladen Swallow was not GvR's project, and it didn't "crash and burn". Unladen Swallow found that their approach was not speeding up Python as much as they had hoped, and the two Google employees were moved on to other projects. The code lives on, and I think people are still doing things with it, although it's clear that PyPy is a better approach.
To me at least, "crash and burn" implies a horrible failure with catastrophic consequences, and Unladen Swallow was considerably less dramatic than that. If you just meant to say "they didn't accomplish their goals", that would be a fair statement.
Python is slow because von Rossum refuses to cut loose the boat-anchor of "anything can change anything at any time".
That was a basic decision way back when, and it would be a profound change to make (the language wouldn't really be Python anymore). I personally don't make much use of this, but supposedly there are some programs that do.
PyPy, the newer Python implementation, uses two interpreters and a JIT compiler to try to handle the dynamism with less overhead. They're making progress, but they need a very complex implementation to do it, and they're many years behind schedule.
There is a very small group of people working on PyPy, and they are doing ambitious things. I'm not overly worried about their schedule.
You failed to mention that PyPy has already achieved great speed when compared to CPython. The latest PyPy is, on average, over 5x as fast as CPython.
http://speed.pypy.org/ [pypy.org]
if a module or class could only be modified from outside itself if it contained explicit self-modification code (like a relevant "setattr" call) most modules and classes could be nailed down as fixed, "slotted" objects at compile time.
This is an interesting idea. But I am not aware of anyone working on something like this.
Claiming that the "slow parts" should be rewritten in C is a cop-out.
I disagree. Way back at the dawn of time, GvR designed Python to make it easy to interface C code, just exactly as this sort of "escape hatch" to help projects that work well but are too slow, and also for libraries.
I think that the ability to link in C code was an important feature that helped Python win hearts and minds. It's slower than C, but at least you knew there is an escape hatch if you wind up having trouble with it.
Except for number-crunching, or glue code for existing libraries, it's seldom done.
Yeah, but that's kind of like saying that except for stop signs and traffic lights, you usually don't use the brake pedal in a car.
There are all sorts of useful libraries that have been glued into Python, and a major part of Python's popularity is that you can use powerful libraries from a convenient and friendly language.
SciPy is a great example: they took ferociously powerful libraries (written mostly in Fortran) and made them usable from Python. I know I for one can get more work done in Python than in Fortran.
As another example, a few years ago I worked on a project to make a DVD player with some fancy features, and we were doing our development in Python. It was fast enough, even on a cheap embedded processor, because all the heavy lifting was being done by C library code. And we got a lot of work done quickly.
(I have a Python program running right now which will run for over a week, parsing the street address of every business in the US into a standard format. The parser is complex enough that rewriting it in C would be a big job. There's no "inner loop".)
This sounds interesting.
Can you run this in PyPy?
Can you fan it out to multiple processes using the multiprocessing [python.org]
Re:007087 (Score:5, Interesting)
People still optimize to assembler.
Very rarely, though. On modern architectures, writing high performance assembly language requires a deep knowledge of how everything fits together. Compiler writers spend a lot of time becoming experts in this field, and a lot more time becoming experts in applying optimisations at different levels of abstraction within an entire codebase during the different stages of compilation/optimisation. Consequently most non-specialist programmers, even those who have done plenty of assembly work in their time, would produce slower final code today by hand-crafting their own assembly language than they would by writing in C and trusting a good compiler to take care of the code generation.
Re:Static vs. Dynamic Typing (Score:4, Interesting)
The point is I can (and have) replaced 100k lines of C, 150k lines of java, or 110k lines of C++ with 10-15k lines of python.
I don't care who you are, its easier and faster to write 15k lines of code than 100k lines of code, significantly faster. Its easier and faster to debug 15k lines of code than 100k lines of code, again significantly. Most bugs are not type errors. Most bugs are logic errors. Sure, is a type error annoying? Yes, but it annoys *me* as a thinking person that I changed the assumed type of a function argument and didn't go change a caller, its not the languages fault that I forgot, just like its not C/C++/Java's fault when you walk off the end of an array and segfault... automated tests find these bugs just as reliably as a compiler, and you have to write automated tests to test logic even in a statically typed language, so its not extra work.
The tradeoff is that my python code might run slower (if the modules I'm using aren't already C which a lot of them are), but vs Java, startup time is significantly faster, and memory usage in python is significantly better, like an order of magnitude at least. vs c/c++ I don't have to deal with pointers, null dereferencing, memory management, or any of those headachy things that a large proportion of developers do wrong, and end up with insecure programs. So... to me at least the choice is clear. Shorter code that is easier to read, easier to debug, quicker to write, easier to maintain, and generally has less bugs, python gets all those things 100% of the time vs C, C++, or Java. The only downside is it might be slow... and if it is, you write a couple hundred lines of C, write all manner of tests around it to make sure it deals with all the pointer stuff correctly, and you're still way ahead of a pure C program.
Want a good example? (Score:5, Interesting)
Civilization 4. Firaxis wanted to make it nice n' moddable. So they scripted a bunch of it out in Python. Makes it real easy for users to edit in the field. However the AI they couldn't do that with. They wanted the users to be able to edit that too, but it was too slow in Python. So they did that in C++ and made it a DLL, and then distributed the sources. Harder to work on than a Python script, but fast enough that the game didn't bog down. The core of the game engine was C++ too and the Python was integrated with Boost.Python.
Works really, really well. Again, the right tool for the job. If they'd tried to do the whole game in Python, it would have been a disaster performance wise (and I'm not even sure if it would be possible, if Python can call on DirectX in the way C++ can). However a mixture worked brilliantly.
Re:007087 (Score:4, Interesting)
Sure, but you'd never write a whole application in assembler just because it was too slow in C. That's what the "Python is too slow" people are doing.
There are even a variety of excellent tools to make interfacing bits of C with Python stupid easy. With Pyrex/Cython you can take your Python code as is and slowly C-ify it until its fast enough for you. Just statically defining a variable or two often makes a big difference, and yes, you can just take your Python code and statically define a single variable using Cython.
Guido's wrong. (Score:2, Interesting)
"Hence what Guido is saying. You can do that "most stuff" part in python, and then write that "some stuff" part as a C module. You can then use that module from python. Thus you get the benefit of both languages."
No you don't. If you had written it in C++ you could natively use C++ objects and libraries fluently.
It takes more skill and system-programming knowledge to deal with the tricky interfaces between the internals of a Python interpreter and an external C++ program. The object structure in Python is obviously alien to the C++ object system. Of course there is always workarounds, but they are inconvenient and require arcane knowledge. Somebody who is an expert at Python internals doesn't think this is hard. Somebody who is an expert at computational algorithms but not compilers does think this is hard, and especially thinks this is something that they really do not have a desire to learn. Me, I have far more interest in spending my time reading a machine learning research paper rather than learning about some crufty programming interface.
The article is a naive cop-out for not specifying a sufficiently good language in combination with implementation techniques (they go together). Guido could try to implement a whole bunch of new libraries in some external Lisp and see how much he likes it versus writing native Python code. It sucks.