Van Rossum: Python Not Too Slow 510
snydeq writes "Python creator Guido van Rossum discusses the prospects and criticisms of Python, noting that critics of Python performance should supplement with C/C++ rather than re-engineering Python apps into a faster language. 'At some point, you end up with one little piece of your system, as a whole, where you end up spending all your time. If you write that just as a sort of simple-minded Python loop, at some point you will see that that is the bottleneck in your system. It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language, because for most of what you're doing, the speed of the language is irrelevant.'"
007087 (Score:5, Insightful)
Title is kinda silly.. as the basic referenced statement is that in some cases python _is_ too slow but that one can work around that using hacks (or a language agnostic component oriented architecture).
As for:
You said that if you trust your compiler to find all the bugs in your program, you've not been doing software development for very long.
It’s not about finding all the bugs, or even many of them. It’s about another layer where a potential bug can be caught. Runtime bugs are the worst kind as they can sit dormant for a while if in a rarely traveled branch. The more checking that can be done at the compile level, the better (imo).
Personally my biggest complaint about python wasn’t on the list: A lot of the (common) libraries out there are poorly documented, inconsistent, buggy, or incomplete.
As a Gentoo user, the python 2/3 thing is also especially annoying. Obviously this isn’t really python’s fault.. but it still gives me a bad taste about python.
That said, this was a great article.. short, to the point, and the answers were pretty good!
Re:007087 (Score:5, Insightful)
And to add to that:
Python Not Too Slow
True. It's not too slow. It's just not fast enough.
Re:007087 (Score:5, Insightful)
as the basic referenced statement is that in some cases python _is_ too slow but that one can work around that using hacks
You completely missed what he said, which is "use the best tool for the job".
Most apps can be written much more quickly in Python than C/C++. If they perform adequately you're done. If there are slow spots, use an extension (it's not a hack) to optimize that tiny part of the app. This has been SOP since compiled languages were first invented; we used to write that last 5% of the app in Assembler. But obviously using C/C++ is more productive than Assembler and usually fast enough, just as using Python is more productive than C/C++ and usually fast enough.
Want a good example? (Score:5, Interesting)
Civilization 4. Firaxis wanted to make it nice n' moddable. So they scripted a bunch of it out in Python. Makes it real easy for users to edit in the field. However the AI they couldn't do that with. They wanted the users to be able to edit that too, but it was too slow in Python. So they did that in C++ and made it a DLL, and then distributed the sources. Harder to work on than a Python script, but fast enough that the game didn't bog down. The core of the game engine was C++ too and the Python was integrated with Boost.Python.
Works really, really well. Again, the right tool for the job. If they'd tried to do the whole game in Python, it would have been a disaster performance wise (and I'm not even sure if it would be possible, if Python can call on DirectX in the way C++ can). However a mixture worked brilliantly.
Re: (Score:3)
Title is kinda silly.. as the basic referenced statement is that in some cases python _is_ too slow but that one can work around that using hacks (or a language agnostic component oriented architecture).
I don't really see it as a hack - it just means that Python can get on and do what it does well - letting the developer focus on the important higher-level stuff, and then if you need to focus on the lower-level to optimise, you do it in something designed for it.
A lot of the (common) libraries out there are poorly documented, inconsistent, buggy, or incomplete.
Really? I have never found this - generally I've found everything Python I've ever used to be really well documented - not that I often need to stray outside the standard library, but Sphinx provides a good toolset for creating really good custom d
Re: (Score:2, Insightful)
No fucking kidding. "Python isn't slow, especially when you rewrite the Python function causing the problem in C..." WTF do these people come from?
Re:007087 (Score:5, Insightful)
This thread is idiocy. The point is, you can write the code in python several times in the same amount of time it takes to write it in C or C++. So if you then spend a fraction of that to optimize it, you write a very small portions in C/C++, you still wind up WAAAAY ahead.
These people come from a place where they understand the value of coders all that it entails. A better question is, where do you come from that you don't have to deal with the reality of programming in the real world?
Re:007087 (Score:5, Insightful)
Also most people don't realize these were the same arguements given back in the 80s for writing stuff in assembly instead of C, and given that people have moved on to C++ since with the same things being said 'If it's too slow in C++ then optimize that routine in C', I would say this explanation is correct... ASSUMING python offers good enough profiling capabilities to pinpoint hotspots in the code so that you CAN optimize them away.
That said as the GGGP stated, gentoo's usage of python in a somewhat slow and half assed manner is frustrating (but if you compare it to ANY rpm based distro it's still faster than heck. Be it a 200 mhz or 3ghz processor. Most of the speed issues seem to be pertaining to sleep states rather than actual code anyways).
Re: (Score:3, Insightful)
People still optimize to assembler. That idea never went away. So pointing out that we've "moved on" to C++ or Java doesn't really mean this kind of issue has gone away.
Some stuff still remains performance sensitive enough that people will go to the trouble of optimizing it at a lower level.
Re:007087 (Score:5, Insightful)
Some stuff still remains performance sensitive enough that people will go to the trouble of optimizing it at a lower level.
And the other side of that coin, is of course, that most stuff is not performance sensitive enough to go to the trouble of optimizing it at a lower level.
Hence what Guido is saying. You can do that "most stuff" part in python, and then write that "some stuff" part as a C module. You can then use that module from python. Thus you get the benefit of both languages.
Re: (Score:3)
Sure but if you save yourself 6 months of coding over the course of a two year project, you've got lots of spare time to figure out those internals. And by the time your next huge project rolls around, you'll just be that much further ahead of the game.
Monoculture is rarely the best option in any large system, including computer systems. It might make things "better" in the short term (by some measure) but if you've used the wrong tool for the job, the cracks will show eventually no matter how theoretical
Re:Guido's wrong. (Score:5, Informative)
It takes more skill and system-programming knowledge to deal with the tricky interfaces between the internals of a Python interpreter and an external C++ program.
Is this experience talking, or guesswork?
Admittedly, I haven't had a need for it myself, but it looks easy enough. And you have plenty of options, too!
1. Extending Python with C or C++ [python.org]
2. ctypes [python.org]
3. Cython [cython.org]
Examples for 1 and 2 [dalkescientific.com]
Example for 3 [perrygeo.net]
Re:007087 (Score:5, Interesting)
People still optimize to assembler.
Very rarely, though. On modern architectures, writing high performance assembly language requires a deep knowledge of how everything fits together. Compiler writers spend a lot of time becoming experts in this field, and a lot more time becoming experts in applying optimisations at different levels of abstraction within an entire codebase during the different stages of compilation/optimisation. Consequently most non-specialist programmers, even those who have done plenty of assembly work in their time, would produce slower final code today by hand-crafting their own assembly language than they would by writing in C and trusting a good compiler to take care of the code generation.
Re:007087 (Score:4, Insightful)
For the most part, the compiler kicks my ass at generating assembly. But every once in a while I find that I have programmed some construct that it's not very good at optimizing, or I've subtly surpassed some parametrized limit after which the compiler decides to stop optimizing. Or maybe I've forgotten to give the compiler the right hints for function visibility, pointer aliasing, or the like. Or heck, maybe I am making some meatspace assumptions that the compiler is not at liberty to make, and I need to find a way to codify the assumptions.
In any case, I agree with you that the solution is almost always "tweak the high-level code until the compiler groks me" and not "discard the compiler and write some assembler"; but I find that the "blind trust" approach to compiler-assisted optimization leads falls apart pretty quickly. Always verify!
Re: (Score:3)
Sorry, but I think your information is at least 5-10 years out of date.
Code generators in compilers obviously aren't perfect, and there are plenty of optimizations they don't do as well as they theoretically could yet. Still, compilers have been using the kinds of technique you describe for years now, even on some specialist chips used for embedding. I doubt many programmers, even those who work in the embedded space, have the skill to consistently outperform a compiler these days.
Re:007087 (Score:4, Interesting)
Sure, but you'd never write a whole application in assembler just because it was too slow in C. That's what the "Python is too slow" people are doing.
There are even a variety of excellent tools to make interfacing bits of C with Python stupid easy. With Pyrex/Cython you can take your Python code as is and slowly C-ify it until its fast enough for you. Just statically defining a variable or two often makes a big difference, and yes, you can just take your Python code and statically define a single variable using Cython.
Re:007087 (Score:4, Insightful)
My father's been in software since the 1950's. There were arguments in favor of octal machine code as being superior to assembler -- on the grounds that if you programmed in assembler, you didn't really understand what the machine is doing.
And all those arguments are...dead. Along with the C-is-slower-than-FORTRAN battles, and the C++-is-slower-than-C, and the bytecode-languages-are-slower-than-compiled.
It turns out that developer productivity is actually more important than almost anything. Sure, there are a couple of niches. But they are small.
Heck, I've got developer customers inspecting over 100K packets per second in C#: they want computer performance, but they need developer performance.
Re:007087 (Score:5, Insightful)
The point is, you can write the code in python several times in the same amount of time it takes to write it in C or C++. So if you then spend a fraction of that to optimize it, you write a very small portions in C/C++, you still wind up WAAAAY ahead.
I have to disagree with that, a bit. C and old-style C++ has a lot of boilerplate allocate/release stuff going on, and yes that really slows you down making sure you got it right. But modern C++ RAII-style auto-everything code isn't like that. I find modern-style C++ and Java equivalent in the time it takes to solve the actual problem.
Python is sometimes faster to write in. If you're doing a lot of parsing, for example, it's great. If the total size of your Python codebase remains small, that really helps. But once you start writing very formal Python where the type of every argument is declared in comments, and error handling being done with exacting precision and logging, and so on, you might as well be writing in C++ or Java.
It's really not so much the language that makes adding code to a large code base slow! It's the need to obsess over geting the details of your thought process recorded in the code once you pass the point that any one person could possibly understand the whole codebase, or it's old enough that the original authors have all left.
Wrting code for small projects is what's (potentially) fast - and Python is great for realizing that potential.
Re:007087 (Score:4, Interesting)
Amen to this:) The biggest issue with scripted languages seems to be that people think simplicity of the language is a substitute for proper design for the solution to your problem. I've been involved in coding a feature where there is a possible solution in both C++ and python; seems that C++ is seen as cumbersome mostly because it requires more forethought and python is then immediately hailed as the victor as soon as it can do a halfassed job of it.
Add to this that most people really can't code C++ and do not understand stl...
Re:007087 (Score:5, Informative)
[Python is sometimes faster to write in.]
Python isn't just fast to write though - it's fast and easy to read and understand, to work with, and it's very easy to keep code clear and well documented. I'm not saying these things are impossible in other languages, but in Python it is effortless, and enocouraged by the language. There are big benefits to using it besides simply 'fast to write'.
Re: (Score:3)
no. If this is the argument, they you should be writing in VB.NET, not C#. VB.NEt is faster and it helps you along a lot more than C# (seriously, I was very surprised C# wasn't as good seeing as its the same bytecode under the covers). So if you really want the programmer performance with a bit of C/C++ (implemented as an inproc COM object, that VB.NET again helps you consume better than C#), you need to be dumping C# today.
I also found that the syntax differences helped. Its easy to get confused about the
Re:007087 (Score:4, Insightful)
You don't need a CS degree to program in Python.
This is true of any language. Degrees don't equal knowledge or experience.
Re: (Score:2, Insightful)
Re: (Score:3)
The fucking language is designed to seamlessly use libraries written in C.
It's hardly seamless. The glue layer is horrible awkward and hopelessly inefficient. Often that gets buried behind a SWIG autogenerated interface that produces truly putrid code. Orders of magnitude more than you would get from writing it by hand, and the hand written interface already sucks.
Re:007087 (Score:5, Insightful)
Because it's significantly faster in terms of development time, or in terms of cost (having the less skilled, cheaper programmers work on it).
It's always been a case of write the critical stuff in C/C++. That's why languages like python and perl make doing so so simple.
Re:007087 (Score:5, Interesting)
Because it's significantly faster in terms of development time, or in terms of cost (having the less skilled, cheaper programmers work on it).
It's always been a case of write the critical stuff in C/C++. That's why languages like python and perl make doing so so simple.
That is heavily project dependent. If you are in a position where you can work with pure Python (i.e. the tedious job of writing wrappers for external libraries is done by others) then yes, you get RAD benefits but that is not chiseled in stone. If you are forced to combine your own C++ components with Python the picture changes. I am involved in a project that has a specialized OLAP engine, written in C++ for speed (no way around it) and a 3D GUI app that runs on top of it which was written in Python for RAD benefits. The biggest single headache we encountered with Python was that yes, you do code the GUI faster in Python, but you loose a lot of development time screwing around with writing Python wrappers for the C++ APIs of the OLAP engine. Plus, there are also endless deployment headaches. It's not as if every customer's machine comes preloaded with Python and even if it does, believe it or not your app may work with one Python distro but it will crash with the another and there are also incompatibilities between versions of Pyhon and you have no way of knowing what version is default on the target machine. You may end up having to bundle your own Python environment with your app for stability. Eventually the GUI team concluded that they were better off time wise if the just wrote the GUI in C++ using QT.
Re:007087 (Score:5, Informative)
As the GP pointed out, if you're skilled enough to write optimized code in C/C++, why fuck around with Python at all?
Because we don't want to spend our time thinking about pointers and how to iterate over things? Because functional programming is actually really nice? Because in Python, you can download some data from the web, analyse it using a machine learning algorithm, plot the results, and install another package on the fly, combining 4 independent packages, and many ideas, in just 50 lines of code.
ctypes is really easy to use and to interface with C or Fortran. I use it a lot, namely for the 1% of the code that takes 99% of the time. The rest is nice OOP and functional.
Re:007087 (Score:4, Informative)
Because we don't want to spend our time thinking about pointers and how to iterate over things? Because functional programming is actually really nice?
You can do all that in C++11 these days, if a tad more verbose.
Note the lack of pointers, and the lambda passed to std::transform.
Re: (Score:2)
Ah, that is a really good troll. :) I am actually quite impressed. :)
Re:007087 (Score:4, Insightful)
It's not about skills, it's about developer time. It generally takes longer to write code in C or C++ that to write something to do the same thing in a higher-level language like Python (C much more so than C++ I would think, especially if you're doing anything involving strings; C really sucks for that stuff; C++ is much better, esp. with a good library like Qt).
Another factor is that you might be working on a larger project, and not everyone is all that skilled. So you can put the Python monkeys to work on some parts, and put the C or C++ people to work on the performance-critical sections, and merge the two together.
Re:007087 (Score:4, Informative)
Python is compiled to bytecode, and said bytecode is then interpreted by Python VM. It's just that this compile cycle happens transparently, and is not explicit as with .java & .class files. It also caches the resulting bytecode in form of .pyc files, and this process can be triggered manually - indeed, many Linux distros have Python packages do that when they are installed. So there's no re-parsing there, and no direct interpretation of AST.
Java is compiled to bytecode, and said bytecode is then JIT-compiled to native code by the VM. The VM does not typically run bytecode directly - or rather, it runs it for a few times, but when it's clear that the function is going to be called a lot, such that overhead of compiling bytecode to native is worth the bother.
Also note that the above description of how Python works only applies to CPython, which is the most popular implementation, but by far not the only one. Python language spec explicitly recognizes the existence of alternative implementations, and language is intentionally designed to cover a wide range of implementation technique (e.g. it does not mandate reference counting and predictable implicit release of orphaned objects, for the benefit of GC-driven implementations, and provides a "with" statement to explicitly release things RAII-style). Hence why we have Jython, IronPython, PyPy etc. PyPy, in particular, implements the full source code -> bytecode -> JIT-compiled native code, similar to Java, and is much faster than CPython.
The real reason why Python will still be slower than Java is because it's a much more dynamic language. JIT-compiler can try to infer types and other constraints, or at least predict the most likely path and optimize for that, but even a single extra check is still slower than no checks.
Re: (Score:3)
The real reason why Python will still be slower than Java is because it's a much more dynamic language.
That's actually a rather bad excuse. Dynamic languages don't need to be that much slower. For instance, have a look at JavaScript V8 or LuaJIT 2 for some incredibly fast dynamic language implementations. If I recall correctly, last time I checked, LuaJIT was about a factor 3 behind optimized C code in the median case on the Computer Language Benchmarks Game [debian.org] (although it seems to be no longer included in that benchmark for some reason).
Re: (Score:3, Informative)
... because given two equally talented developers, the one working in python will literally run circles around the guy working in C/C++... and in the real world developer time = money.
The fact is that interfacing with C libraries in Python is already trivial. Furthermore modern tools like Cython make it EVEN EASIER!
So you take your code, profile it, decorate the most time-intensive portions to be compatible with Cython (trivial for most applications) -or- interface it with your C library. This way, you ge
Re:007087 (Score:5, Funny)
the one working in python will literally run circles around the guy working in C/C++
Pedant police here. You are under arrest for abusing the word 'literally'. Hand over your poetic license and step away from the keyboard.
Re: (Score:3)
the one working in python will literally run circles around the guy working in C/C++
You should fire that python guy. Not only is he being unproductive, it's very disruptive and will likely really hurt the productivity of your other staff members.
Oh, and the c/c++ thing makes you look like an idiot... Wait ... are you the guy running around that other developer?
Re: (Score:3, Insightful)
You must be retarded because it was already explained to you above - which you even acknowledged.
In the real world, programming costs dollars. You can write something in python for 1/8 - 1/2 the cost (cost is a function of time) and optimize a tiny section of code or you can write it in C++ and get nearly identical performance. These need not even be the same programmers; which means the opportunity for additional savings is there. Furthermore, contrary to the assertion, this is not black art programming. T
Re: (Score:3)
I take it that you never use Windows logon batch scripts. I mean, if you are skilled enough to implement "net use E: \\server\share" in C++ in a few hundred lines of code, why the hell wouldnt you? It will run a few microseconds faster!
Re:007087 (Score:4, Informative)
As most things in life do, code usually follows the Pareto distribution: 80% of the time is spend in 20% of the code. If Python is fast enough for, let's say 90% of your code, and you are much more productive in Python than C, then writing most all the code in Python first, and replacing the bits and pieces that are too slow for you with C functions, is much more efficient use of your time than writing everything in C.
Also consider that sometimes you have to go in fishing expeditions for the correct algorithm to do what you are doing. Doing so in Python, with the speedup in iterative design that that carries, and even if that once you find the most efficient algorithm for your problem you implement it in C, you will have had spend the same time as before, but knowing all the ways you can't do it, and have arrived to an at least nearly optimal solution.
Most of the time you don't need that much speed. When you do, you have to have the right algorithm and the right language. I also put forth that Python has a lot of modules that are already written in C, so you take advantage of existing optimized code that you don't have to write.
Re: (Score:3)
Yeah but on actually using the language this isn't an issue. It certainly works for me.
Re:007087 (Score:5, Insightful)
I'm always wary of people who say a language "sucks". Each language was designed the way it as for certain reasons. You may not agree with those reasons but they are there.
As it turns out I enjoy using Python quite a bit and it helps me get my work done nice and quickly, which mainly involves web code and associated sales processing. I could sit there and worry about what it can't do but what it can do is just fine.
Obviously the bash comparison is nonsensical. Bash is hardly the same kind of language as Python and I think you know that.
Re: (Score:3)
I'm always wary of people who say a language "sucks".
Well, if you don't know what I mean when I say "bash sucks" then you haven't written much bash. And by the way, it is in many cases the best tool for the job, but that does not mean it does not suck.
Re:007087 (Score:4, Funny)
Re: (Score:2)
But python brings some things to the table that other languages don't have, like some of those libraries that, as you said, are not well documented, but most of the time do what you want and you don't have to worry about memory management, pointers, etc. while having access to dictionaries out of the box.
I'm all for writing the program in whatever language you feel more comfortable on and just do the heavy duty stuff in c++ daemons or something like that.
Re:007087 (Score:5, Insightful)
Python, apart from the inconsistent standard of the libraries that the GP mentioned, is brilliant for the 'glue code' that holds small to medium sized programs together. Ruby may have the edge for organising larger programs.
Rewriting a relatively small Python program in C++, one that does a fair bit of file handling, string chopping about etc, can make it into a relatively large program. Doing it in C is asking for pointer problems. I've been programming in C from before C++ existed, C++ since then and Python for about 5 years.
I am quite capable of writing the glue code successfully in any of the languages, but to do it efficiently I'd choose to do exactly as TFA says - write the main bulk of the code in Python( (or Ruby, or some other higher level language), then measure it to find the real slow bits (you wouldn't optimise until you have measured where the problems REALLY are, would you?) and rewrite just those bits in a lower level, faster language.
Of course, once you see where the speed issue lies, it's just as likely you can find an algorithmic improvement that fixes it 'enough' anyway...
Re:007087 (Score:5, Interesting)
I have been coding in C for quite a while. Everything from simple user programs through kernel code, drivers, and firmware. I find that python is quite liberating. It allows me to accomplish a lot more faster and have it tend to just work. I do perform the analysis and occasionally re-implement a performance critical function in C, but I find that it's surprising how little that actually improves most situations, not because of a deficiency in Python, but because it wasn't actually all that slow.
Yes, I could implement the cool handling of various data types Python has in C, but doing so will produce something that looks a LOT like a poorly tested and less optimum implementation of Python.
Agreed. (Score:5, Insightful)
Now that that is settled we can get back to the real problem with python: Type errors.
Re:Agreed. (Score:5, Informative)
Python is strongly typed. Maybe you mean statically typed.
Re: (Score:2)
That was covered in the article as well (kinda) .. with the same kind of "just do more testing" type answers you hear from the PHP guys.
Personally I like my languages strongly typed, with as much idiot proofing and compile time checking as I can get and still have a usable language. Not as a substitute for testing/QA as the article would imply.. but as an additional layer.
Well said. Developers should think of static typing and strong typing as testing built right into the code -- testing your boss can't make you skip writing in order to meet a deadline, because if you have type errors, your code will not even compile.
Re: (Score:3)
You can do a few interesting things with duck typing that don't work very well in a static type system. My object may not be a foo, but it does have a bar method that needs calling and that's all that matters. If it bothers you, just type check it: if not has_method(var, 'bar'): or just catch the exception.
It's better than C++ where you cast a pointer and the code happily treats random gibberish as the type you cast to.
That said, a static analysis tool would be a nice thing to have.
Re: (Score:3)
As someone who's worked with Python for years both in personal projects and professionally, I've never had an issue with that.
Because your anecdotal experience means that the debate is over. Your single, biased data point is plenty to draw sweeping conclusions.
I hope you write better program logic, because your logical thinking skills are definitely lacking.
Maybe python attracts more intelligent programmers than the dime-a-dozen .Net / Java programmers being pumped out of university.
Oh. You're one of those "I'm smarter than everyone else" types.
Re: (Score:3)
Thousands if not millions of people get by just fine with Python.
Yes, and the same goes for C/C++ or Java. The code for the moon landings was also written in assembly. The simple fact is that you can get the job done in any language, but that doesn't mean there aren't advantages and disadvantages to particular languages.
Logically Logical Logic (Score:5, Funny)
"It is usually much more effective to take that one piece and replace that one function or module with a little bit of code you wrote in C or C++ rather than rewriting your entire system in a faster language"
Ahh -- yes, I see, so I should write my Apps in Python, except where they need to be rewritten in C/C++ because that will run faster than when written in Python, but Python is not slow when you rewrite portions -- so don't rewrite in a faster language because Pyton is fast enough.
Alrighty then.
Re:Logically Logical Logic (Score:5, Insightful)
Ahh -- yes, I see, so I should write my Apps in Python, except where they need to be rewritten in C/C++ because that will run faster than when written in Python, but Python is not slow when you rewrite portions -- so don't rewrite in a faster language because Pyton is fast enough.
Alrighty then.
Essentially yes, that's it exactly. It's a lot simpler to write a 5000 lines of python and 300 lines of C than it is to write 20,000+ lines of C. Plus Python manages most of the memory management for you so you have less chance of memory leaks. I would argue that the reduction in bugs memory bugs and more maintainable code would justify saying that one should use two languages in this case. It's not a matter of which is better overall, it's that python is easier to read, whereas C is faster. Use both where their benefits are most powerful.
Re: (Score:3)
It's also possible to write better x86 than a C/C++ compiler, so I hope you're writing everything in Assembly.
Re:Logically Logical Logic (Score:5, Informative)
Yes, that is correct. You should write your apps in Python.
Your libraries, you should write in Python first, because it is also a great prototyping language. If they work fine (which they will in most cases) you have saved yourself a bunch of time. If they are too slow, you have saved yourself a bunch of time by fixing algorithmic bugs in a flexible language like Python. It is now trivial to convert it to bug-free C or C++.
Re: (Score:3)
So you can write all your code in Python, find through profiling which small pieces of code are hot/heavily used, and replace that with C code.
Or write it a bit smarter. There was a poster on one forum that had a python program doing some image processing. Basically finding the nearest pixel from center within a certain variance of a color.
It took ~60 seconds on a test image he had, and he had found out that the majority of the time was spent doing color convert (RGB to LAB). Removing that step ran the code in 4 seconds.
When I stumbled over the thread, people had been improving parts of it for 3 days already, and it was down to ~55 seconds run tim
Re: (Score:3)
The Blender project has some pretty nice boolean operations written in Python, but they are history because of performance issues. And it would seem, they were not significantly less buggy than the buggy C code they attempted to replace.
Python: Not Much Worse Than Ruby (Score:5, Funny)
Van Rossum: Python Not Much Worse Than Ruby
"Python creator Guido van Rossum discusses the prospects and criticisms of Python, noting that critics of Python should supplement with Ruby rather than re-engineering Python apps into a better language."
Re: (Score:2, Insightful)
That's the problem with Python. Ruby is expressive and object-oriented, C is fast. Python is neither.
Re: (Score:3)
Re: (Score:3)
Corollary:
Van Rossum: Python Not Much Worse Than Perl
Just had to be said.
Kinda digging Python (Score:5, Insightful)
if(a > 0) { return a + 1; } else { return a -1; }
for the simple stuff. I do really like things like being able to return multiple values from a procedure, etc., but Python seems more useful for rapid prototyping rather than anything else.
Re: (Score:2)
Being able to return multiple values is nice (although you can do that in C++ with a Pair template object, it isn't frequently done). Just remember there's a lot of pitfalls. For example, if you use the frequently used for loop replacement
for x in range(len(somearray)):
you're actually doing this in C
int size = length(somearray); //do loop body on j}
int array = new int[size];
for (i = 0; i size; i++) {array[i] = i;}
for (i =0; i size; i++){ j = array[i];
If size is big, you're hosed on memory and time.
Now t
Re:Kinda digging Python (Score:4, Informative)
If you do need to know the index, you should write
It pays off to spend some time learning not only the syntax when you learn a new language, but also often used idioms in that language.
Re: (Score:2)
Re: (Score:3)
Nitpick: Python can't actually return multiple values; it's just that the comma create a single value - a tuple - that aggregates the values passed to it.
When you do "return a, b", you're actually creating a tuple (a, b) and returning that.
In practice, it's mostly irrelevant, though.
Re: (Score:3)
Actually Python has found quite a niche in web-based applications. There are a fair number of large Django-based sites out there. Also RedHat writes most of their system utilities, including their installer, Anaconda, in Python. Of course since Guido works for them, this is understandable. But RH was using Python in RH system utilities back in the Python 1.5 days. Virt-manager is written in Python with C-based modules to interface with libvirt. Anaconda itself uses c-based modules for hardware interfa
Re: (Score:2)
return a+1 if a>0 else a-1
Just starting out, so they didn't teach us that syntax. We've been shown:
if (a > 0):
return a + 1
else:
return a - 1
Pretend that there are 4 spaces before the 2nd & 4th lines, since Slashcode doesn't recognize non-breaking spaces.
Re:Kinda digging Python (Score:5, Informative)
return (a>0)?a+1:a-1;
Tertiary operator FTW!
usually? (Score:5, Insightful)
I have a lot of experience in code optimization, and I would dispute this generalization. "often" is a lot more realistic than "usually". The most common thing I see is where one particular segment of an operation is coded by someone that doesn't understand their O's and is doing something like multilevel lookup loops instead of a hash table. Fundamental mistakes in algorithm choice are the biggest "HERE is the biggest problem" issues I find.
Once you're past the stupid implementation mistakes, it goes just slightly in favor of "it's a little bit of everything" land. Something running significantly slower in one language than another often boils down to the coder not understanding how to make things scale in the chosen language. I can make C move slower than BASIC if I want to. Sometimes it's just knowing how the compiler is going to react to your structures. Little things like "roll up the loops when coding in VB" can produce an order or two of magnitude in speed improvement, and if you don't realize this you may think you're comparing identical implementations when you're not. "this language sucks!" often translates into "I don't know how to do it so it runs fast!"
My last project was reduced from 23 hrs per run to 21 minutes by a small but complex change in implementation. From there, getting it down to 4 minutes required a LOT of little changes all over the place, to nickel-and-dime it down. I'll trade you my "guy that knows how to recode it in C" for your "guy that knows how to code, and REALLY knows his compiler" any day.
Re: (Score:3)
Re: (Score:3)
Personally (Score:5, Informative)
As someone simulating fluid-structure interaction with a number of constituent models and a lot of finite element (i.e. big matrix problems; using FEniCS - fenicsproject.org), using Python makes my overall quite-long algorithm much easier to flick through. Invaluable for debugging the theory as well as the implementation. FEniCS' Python interface ties into the standard C/C++ libraries using SWIG and, in simple cases, saves me working in C++. Very clear, well-written C++ is great for this application but I find it takes considerably longer to write than clear Python.
When I hit a more intricate problem, I realized I was going to have to solve a series of FE matrices by hand (with PETSc, written in C). It turned out to be pretty straightforward to pick up SWIG, write a short module in C and a Python interface. Done! Particularly useful as I believe getting FEniCS and petsc4py to play well is tricky.
So, I'd agree - having written a C++ version of my (simpler) problem and a Python/C version of the complicated one, the latter was definitely easier, and all the rate-limiting stuff is in C anyhow.
Doubt it would be true for every situation but +1 from an FE perspective.
Purists! (Score:4, Insightful)
To bend a cliche: Purists! Can't live with 'em, can't live without 'em!
Seriously, we live in an era when programmers are no longer bound to the use of a single language for an entire project, as was the pragmatic case once upon a distant time. Why not just use the language for each module which best suits the need? If performance outweighs simplicity of code management, then use the better performing language for that module. No language is perfectly suited for all goals, so own your chosen criteria and don't 'blame' a language creator for having different criteria.
Metaphor (Score:4, Insightful)
Arguing about whose interpreted scripting language is slower is like arguing about whose rich delicious cheesecake is less fattening. When you eat the cheesecake, you accept the tradeoff of tastiness for five minutes off your total lifespan.
Python's problem (Score:5, Informative)
Re: (Score:3)
Multithreading is only a problem in CPython. Other interpreters like Jython, IronPython or PyPy all have facilities for multithreading.
Isn't this what all the scripters do? (Score:3)
Don't most users of these scripting languages (the good ones anyway) profile and write the speed-critical sections in C or C++ anyway? That's not Python specific. It's not even specific to scripting languages. It's the same thing that C programmers do when they use inline assembly. It's like this all the way down the line. You start with rapid development at a higher level, then profile and optimize what you need.
It's still too slow, despite what he says. (Score:5, Informative)
Says the guy whose whole life is tied up in the language, and whose project, at Google, to speed it up, crashed and burned. [wikipedia.org]
Python is slow because von Rossum refuses to cut loose the boat-anchor of "anything can change anything at any time". The straightforward implementation of Python, CPython, boxes all numbers (everything is a CObject, including an int or a float) and looks up functions, attributes, and such in a dictionary for every reference. And only one thread is allowed to run at a time. This allows one thread to dynamically patch the objects and code of another thread. Which is cool, but useless. 99.99+% of the time, there's no need for a dynamic lookup. Most program dynamism is shortly after program startup - once things are running, they don't change much. If, sometime shortly after startup, the program said "OK, done with self-modification", at which point a JIT compiler did its thing, the language would be much faster. But no. That's "un-Pythonic".
PyPy, the newer Python implementation, uses two interpreters and a JIT compiler to try to handle the dynamism with less overhead. They're making progress, but they need a very complex implementation to do it, and they're many years behind schedule.
Python, as a language, is very usable. But it's too slow for volume production. That's not inherent in the basic language. Python could remain declaration-free if there were just a few more restrictions on unexpected dynamism. By this is meant ways the program can change itself that aren't obvious from looking at the source code. For example, if a module or class could only be modified from outside itself if it contained explicit self-modification code (like a relevant "setattr" call) most modules and classes could be nailed down as fixed, "slotted" objects at compile time. The other big win is using enough type inference to decide if a variable can always be represented as a machine type (int, float, char, bool, etc.). That's a huge performance win.
Claiming that the "slow parts" should be rewritten in C is a cop-out. It makes the program more fragile, since C code can break Python's memory safety. Except for number-crunching, or glue code for existing libraries, it's seldom done.
(I have a Python program running right now which will run for over a week, parsing the street address of every business in the US into a standard format. The parser is complex enough that rewriting it in C would be a big job. There's no "inner loop".)
Re: (Score:2)
(I have a Python program running right now which will run for over a week, parsing the street address of every business in the US into a standard format. The parser is complex enough that rewriting it in C would be a big job. There's no "inner loop".)
This would be a good application for Perl. Perl handles regexps about 4 times faster than Python.
Re:It's still too slow, despite what he says. (Score:5, Interesting)
Says the guy whose whole life is tied up in the language
That's fair.
and whose project, at Google, to speed it up, crashed and burned.
That's completely wrong. Unladen Swallow was not GvR's project, and it didn't "crash and burn". Unladen Swallow found that their approach was not speeding up Python as much as they had hoped, and the two Google employees were moved on to other projects. The code lives on, and I think people are still doing things with it, although it's clear that PyPy is a better approach.
To me at least, "crash and burn" implies a horrible failure with catastrophic consequences, and Unladen Swallow was considerably less dramatic than that. If you just meant to say "they didn't accomplish their goals", that would be a fair statement.
Python is slow because von Rossum refuses to cut loose the boat-anchor of "anything can change anything at any time".
That was a basic decision way back when, and it would be a profound change to make (the language wouldn't really be Python anymore). I personally don't make much use of this, but supposedly there are some programs that do.
PyPy, the newer Python implementation, uses two interpreters and a JIT compiler to try to handle the dynamism with less overhead. They're making progress, but they need a very complex implementation to do it, and they're many years behind schedule.
There is a very small group of people working on PyPy, and they are doing ambitious things. I'm not overly worried about their schedule.
You failed to mention that PyPy has already achieved great speed when compared to CPython. The latest PyPy is, on average, over 5x as fast as CPython.
http://speed.pypy.org/ [pypy.org]
if a module or class could only be modified from outside itself if it contained explicit self-modification code (like a relevant "setattr" call) most modules and classes could be nailed down as fixed, "slotted" objects at compile time.
This is an interesting idea. But I am not aware of anyone working on something like this.
Claiming that the "slow parts" should be rewritten in C is a cop-out.
I disagree. Way back at the dawn of time, GvR designed Python to make it easy to interface C code, just exactly as this sort of "escape hatch" to help projects that work well but are too slow, and also for libraries.
I think that the ability to link in C code was an important feature that helped Python win hearts and minds. It's slower than C, but at least you knew there is an escape hatch if you wind up having trouble with it.
Except for number-crunching, or glue code for existing libraries, it's seldom done.
Yeah, but that's kind of like saying that except for stop signs and traffic lights, you usually don't use the brake pedal in a car.
There are all sorts of useful libraries that have been glued into Python, and a major part of Python's popularity is that you can use powerful libraries from a convenient and friendly language.
SciPy is a great example: they took ferociously powerful libraries (written mostly in Fortran) and made them usable from Python. I know I for one can get more work done in Python than in Fortran.
As another example, a few years ago I worked on a project to make a DVD player with some fancy features, and we were doing our development in Python. It was fast enough, even on a cheap embedded processor, because all the heavy lifting was being done by C library code. And we got a lot of work done quickly.
(I have a Python program running right now which will run for over a week, parsing the street address of every business in the US into a standard format. The parser is complex enough that rewriting it in C would be a big job. There's no "inner loop".)
This sounds interesting.
Can you run this in PyPy?
Can you fan it out to multiple processes using the multiprocessing [python.org]
Static vs. Dynamic Typing (Score:5, Insightful)
I was a little bit disappointed by Guido's response regarding static vs. dynamic typing:
InfoWorld: You talked about the arguments for and against dynamic typing. You said that if you trust your compiler to find all the bugs in your program, you've not been doing software development for very long. So you're satisfied with Python being dynamic?
Van Rossum: Absolutely. The basic philosophy of the language is not going to change. I don't see Python suddenly growing a static subdivision or small features in that direction.
Proponents of static typing do not claim that compilers, combined with languages that use static typing, will find all the bugs in your program. This is nothing more than Infoworld erecting a straw man and Guido knocking it down.
However, static typing does make a huge number of potential errors stick out like a sore thumb (the compiler will refuse to compile the code, and will emit appropriate error messages).
Some people (rightfully) argue that dynamic typing makes for shorter, prettier, easier code.
Some of us believe the primary concern should be correctness, and that shorter, prettier, easier code are secondary concerns -- almost always. People should think about this every time their computer crashes, or an application crashes, or something is acting up and needs to be rebooted, or they get a virus through no fault of their own, or their data gets corrupted.
Will users be thinking, "Gosh, this sucks, but I'm sure glad the programmer used a dynamic language, because it made it easier on him (the programmer)."? No, they'll be thinking, "Damn buggy programs! I just lost X (hours,minutes,seconds) of work, and now I'm frustrated!" Programming languages are a means to an end, not an end in itself. Don't be a self centered developer: the fruits of your labor are for users, not so you can write the code equivalent of poetry.
Not to mention, statically typed languages allow for easy refactoring possibilities that make it possible to fix all sorts of serious issues, including architectural ones, with reasonable effort expended. Dynamic languages, while they have made some progress in the area of refactoring, are really in the dark ages here.
I know dynamically typed programming languages are the hotness right now, and I'm sure my opinion will be hammered relentlessly, but I do ask that if you disagree, don't mod me down, but instead, bring forth a reasonable argument for a different position. This should not be a popularity contest, where the loser is not heard, no matter what side the loser is on.
Re: (Score:3)
As someone who's been working on a fairly large scale application suite in Python for the last 4 years I would say that dynamic typing is both a blessing and a curse during development. It certainly makes writing and reusing code easier to an extent, but it can make some things more difficult especially when you have code that deals with objects of several different types and there's the potential to get things confused.
However, I could probably count the number of times I've seen a part of our system fail
Re:Static vs. Dynamic Typing (Score:4, Insightful)
However, static typing does make a huge number of potential errors stick out like a sore thumb (the compiler will refuse to compile the code, and will emit appropriate error messages).
Absolutely. Do you really want to be wasting company resources tracking down bugs that could easily have been found by the compiler in a statically typed language?
I've never really been a fan of dynamically typed languages. What exactly is the point? To save a few seconds on declaring the type of a variable? You may save a few seconds, but most time tends to be spent debugging code, not writing code. You end up saving seconds in the writing stage and losing hours in the debugging stage. If you really need a variable to be able to change types, many statically typed languages have some kind of built-in variant/object type that can do exactly that if you really need it.
Re:Static vs. Dynamic Typing (Score:4, Interesting)
The point is I can (and have) replaced 100k lines of C, 150k lines of java, or 110k lines of C++ with 10-15k lines of python.
I don't care who you are, its easier and faster to write 15k lines of code than 100k lines of code, significantly faster. Its easier and faster to debug 15k lines of code than 100k lines of code, again significantly. Most bugs are not type errors. Most bugs are logic errors. Sure, is a type error annoying? Yes, but it annoys *me* as a thinking person that I changed the assumed type of a function argument and didn't go change a caller, its not the languages fault that I forgot, just like its not C/C++/Java's fault when you walk off the end of an array and segfault... automated tests find these bugs just as reliably as a compiler, and you have to write automated tests to test logic even in a statically typed language, so its not extra work.
The tradeoff is that my python code might run slower (if the modules I'm using aren't already C which a lot of them are), but vs Java, startup time is significantly faster, and memory usage in python is significantly better, like an order of magnitude at least. vs c/c++ I don't have to deal with pointers, null dereferencing, memory management, or any of those headachy things that a large proportion of developers do wrong, and end up with insecure programs. So... to me at least the choice is clear. Shorter code that is easier to read, easier to debug, quicker to write, easier to maintain, and generally has less bugs, python gets all those things 100% of the time vs C, C++, or Java. The only downside is it might be slow... and if it is, you write a couple hundred lines of C, write all manner of tests around it to make sure it deals with all the pointer stuff correctly, and you're still way ahead of a pure C program.
Re:Static vs. Dynamic Typing (Score:4, Informative)
Will users be thinking, "Gosh, this sucks, but I'm sure glad the programmer used a dynamic language, because it made it easier on him (the programmer)."? No, they'll be thinking, "Damn buggy programs! I just lost X (hours,minutes,seconds) of work, and now I'm frustrated!" Programming languages are a means to an end, not an end in itself. Don't be a self centered developer: the fruits of your labor are for users, not so you can write the code equivalent of poetry.
It's true that static type checking can catch bugs for you. That's why, in C, I'm rigorous about declaring const for anything that ought to not change; I like it when the compiler catches a bug for me.
But the world isn't as binary as you make it out to be. It's not a choice between poetry-like code or more correct code; there is more to it than that.
Python's "duck-typing" can let you write one function where you would need to write several in C or C++. As a contrived example:
def add5(x):
return float(x) + 5.0
It doesn't matter what you pass to this function, as long as it can be converted to a float. In C++ you would have to write multiple versions of this with different type signatures. In C you would have to write multiple versions of this with different names! And it would truly suck if just one of the various versions had a bug in it; in Python, with just one function, you have one place to look for bugs.
(You could probably do this contrived example with templates in C++, or even with a macro in C. And this is a useless example. But I think it makes the point, and real examples need not be as useless and trivial.)
Not to mention, statically typed languages allow for easy refactoring possibilities that make it possible to fix all sorts of serious issues, including architectural ones, with reasonable effort expended. Dynamic languages, while they have made some progress in the area of refactoring, are really in the dark ages here.
I have no idea what you are talking about here. Dynamic languages, which impose fewer limits on what you can do, have some sort of disadvantage over statically-typed languages for refactoring? How does that work?
And let me counter that with an example. In Python, you are encouraged to simply read and write member variables directly, rather than writing getter and setter functions. But if you ever need to "hook" the getting or setting, you can write a property descriptor [gomaa.us] and run a getter or setter function, without needing to rewrite any code; the property descriptor does an implicit function call for you can you can do any checking you need.
And finally, every language has its best practices. In the Python community, self-tests are strongly encouraged. Yes, with Python, type errors aren't caught until runtime; but you can easily add self-tests that exercise the code and catch the type errors pretty much immediately after you introduced them.
http://docs.python-guide.org/en/latest/index.html [python-guide.org]
No language is perfect, but in my experience Python hits a sweet spot. I can get a lot of work done in a few lines of Python, I can write those lines quickly, I can read them when I go back later, and I enjoy the coding more than C. I don't think my Python code is buggier than my C code per line of code, and in Python I write so many fewer lines of code I think I come out ahead on bugs.
steveha
Re: (Score:3)
All I can say is you've obviously never built a large scale system in either a static or dynamic language... The number 1 correlation between bugs and code is the number of lines of code. More lines of code == more bugs. Always, every time. The fact that I can write the same thing in python in 1/5 or 1/7th the code vs Java or C++ means on average I'll have less bugs.
As someone who's spent the last year rebuilding in python the middle and frontend tiers of a large system that integrates a large number of
Re: (Score:3)
Static typing can be shoehorned on Python by an external tool - it already has everything necessary in the syntax to provide type annotations, you just need something to actually enforce them. E.g. you can write, today [python.org]:
but it doesn't really mean anything special to Python itself. It will stuff those things up into __annotations__ property of the function, and so a decorator can be written that wraps the function and verifies types at run-time. Fo
Re: (Score:3)
You can't just make general statements about dynamic x static types. The problem is that altought they are well defined concepts, their advantges and disadvantages depend more on the choosen idioms than on any other thing, and the idioms change a lot from one language to the other.
When you ask a Java developer what he thinks, he'll likely answer you that static types are essential for architecturing a system, and somebody would be crazy to go without them.
If you ask a Perl developer, why he should have the
Re: (Score:3)
Well, as a C++/python developer I spend 90% of my time in C++ wrestling with the obscure syntax and memory management to tell the compiler what I want to do.
C++ evolved into a monstrosity. I share your dislike of C++. I won't touch it unless I absolutely have to.
Shorter prettier easier code is less likely to be buggy.
In general, I agree; however, when it comes strictly to static typing vs. dynamic typing, I think giving up static typing for a little less code is a false economy.
Decent error handling makes code robust. Ack, I ran over the end of my std::vector. Better hope I tested for that. Ack, I ran over the end of my list. Lucky python throws an error for me.
I, too, use a language that does bounds checking for me (that language is not C or C++, is also statically typed, and outperforms Python by miles). I will only "drop down" into C if I need absolute top performance -- which is pretty rare, the
Donald Knuth (Score:4, Informative)
Jon Louis Bentley ``Writing Efficient Programs" (Score:2)
Look, someone has to say it... (Score:4, Insightful)
Integrated multi-language solutions are teh suck.
I know that Python is much better than a lot of other languages for integrating C/C++ code. But in the end, if you're doing production systems, you'll end up getting bitten by some unforeseen incompatibility caused by some upgrade somewhere.
It will happen.
Honesty (Score:3)
It's nice when the proponents of a piece of technology can be honest about its shortcomings and acknowledge them, rather than trying to sell their technology. It's the honest response vs. the fanboy response.
Problem: "It's too slow."
Honest response: "Yeah, sometimes it is slow. Here are the design compromises that make this slowness necessary. Furthermore nobody has volunteered the manpower to make it faster. If you need something faster, try x."
Fanboy response: "Yeah well, anytime I try to do something, it's fast enough."
Problem: "There's practically no static typechecking."
Honest response: "You're right, there isn't. There are ways you can deal with that, such as writing tests and thinking carefully about your code. The lack of static typechecking allows us to build a language that is better in this way and in that way. If you want static typechecking, try language x. Certainly there are advantages and disadvantages to the way we did things, but it best suited our objectives and if your objectives are aligned with ours, consider using our system."
Fanboy response: "Yeah well, if you're relying on a compiler to catch all your bugs, you're in trouble."
I know there are a lot of C++ haters here but having read some of what Bjarne Stroustrup has written, he typically offers honest responses, not fanboy responses. I was impressed with an article on the Arch Linux wiki once for the same reasons, as it gave an honest response to "it takes too long to install" (which was something like "yeah, it takes awhile, which has advantages and disadvantages. Try Ubuntu for quick installs.") rather than a fanboy response ("quick installs, who wants that crap anyway, it will just give you a junk system.")
To be fair to GVR this was a short informal article; it may not be fair to expect him to expound in this kind of a format.
Re: (Score:2)
Re: (Score:2)
Duh!!
It's not Chrome, it's Chrome.py
Re:Language Philosophies (Score:4, Informative)
Strictly speaking, the language itself shouldn't have any effect on how fast it executes, it's the implementation that really matters.
This is nonsense.
Language syntax has a huge impact on how hard or easy it is for a compiler (ahead-of-time, just-in-time, or hybrid) to produce fast native code.
If that effort is too large, in terms of development effort and/or compiler analysis effort, you will simply never see a compiler written for those kinds of languages that produces fast executables. This is the reality.
So, in pragmatic terms...yes, some languages are slow.
Re:world needs a better high performance language (Score:5, Insightful)
As far as high performance languages go, we have:
FORTRAN: King of the performance hill, but so annoying to use that nobody really does outside some scientific circles.
C++: This is what everybody uses to write high performance applications, but it's a mess of special cases and annoying syntax and megabyte-long error messages from deeply nested templates.
We need a modern language, with things like functions as first class objects and introspection, but with the performance and "to-the-metal" nature of C++ when you care about designing for optimal cache efficiency and so on.
This is entirely true. What C++ does is excellent. The standard libraries are great - self-resizing arrays and sane strings at bare-metal speeds (if used with just a bit of skill). All the common algorithms . But the C baggage is really a problem.
There's a lot of syntax and just "tricks" that are needlessly complex becuase of the history. The learning curve is not just steep, but pointlessly steep. The level of control you get does not require the level of complexity thrown at you.
And the worst is - people still write C-style code in C++! Because C-style coding is obious in the language, and RAII is not, you still see people thinking exceptions are bad, and programming like it's 1989. Because template syntax is the worst macro langage ever, you don't dare use templates outside of seldom-changing library code.
All of the downsides of C++ are fixable with a from-scratch language with the exact same feature set, but no legacy syntax.
Re: (Score:3)
you still see people thinking exceptions are bad
Well, there are two things at work here:
Re: (Score:3)
I use C++ for the STL (despite the awful syntax) and for better encapsulation. Boost scoped_ptr is nice too. Virtual functions are a nice alternative