You Can Now Profile Python Using Arm Forge (arm.com) 103
Python "is often described as being slow when it comes to performance... But is that truly the case?" writes Patrick Wohlschlegel, Arm's senior product manager for infrastructure and high-performance computing tools.
Slashdot reader igor.sfiligoi writes: Effectively profiling Python has always been a pain. Arm recently announced that their Arm Forge is now able to profile both Python and compiled code.
It's available for any hardware architecture, Wohlschlegel writes, adding that developers "typically assume that most of the execution time is spent in compiled, optimized C/C++ or Fortran libraries (e.g. NumPy) which are called from Python..."
"How confident are you that your application is not wasting your precious computing resources for the wrong reasons?"
Slashdot reader igor.sfiligoi writes: Effectively profiling Python has always been a pain. Arm recently announced that their Arm Forge is now able to profile both Python and compiled code.
It's available for any hardware architecture, Wohlschlegel writes, adding that developers "typically assume that most of the execution time is spent in compiled, optimized C/C++ or Fortran libraries (e.g. NumPy) which are called from Python..."
"How confident are you that your application is not wasting your precious computing resources for the wrong reasons?"
Slashvertisement (Score:5, Informative)
You're supposed to mark the sponsored content....
Re: (Score:2)
I realize there’s a tendency here on Slashdot to label anything remotely commercial as a “slashvertisement”... but this one really is. The link goes to a sales spiel from arm’s software group for one of their products.
Re: Slashvertisement (Score:3, Informative)
Itâ(TM)s not even an innovative product. Python comes with two different profilers, and thereâ(TM)s a plethora of other profilers and guis that work with it.
Also, the summary itself is largely excerpts from said sales pitch.
Why waste time profiling Python? (Score:1)
It's like getting a rusted out 73 Chevy detailed.
If you cared about performance you wouldn't have used Python to start with.
It's slow (Score:3)
Python "is often described as being slow when it comes to performance... But is that truly the case?"
Yes, in fact yes, it is. That is not why we use Python.
Re: (Score:3)
Re: (Score:2)
What developer assumes that most of the execution time is spent in compiled libraries when the program is written in an interpreted language? I mean, if I write a c++ library to call from my Python program, it's because I know it's gonna be fast...
Re: (Score:2)
Re: (Score:2)
It depends tremendously on the working environment. The "Python for Windows" package from pypi.org take shocking amounts of time to even begin running the simplest python scripts. The Python built into CygWin, however, has much more reasonable behavior. As best I can tell, it's the underlying filesystem access components, which are much better integrated with CygWin.
Re: (Score:1)
Re: (Score:2)
And no Inline:C (or dozen other languages) that I know of. I'm no python wizard, I learned other languages that get 'er done for me.
As the AC didn't quite manage to point out, all interpreted languages are always going to be slower than something that compiles to native machine code except for very special cases of misuse. But you gain things that might be worth it to save programmer time that can't be done -
Re: (Score:3)
This is for parameter sweeping a fusion reactor I'm working
Re: (Score:2)
If you want to write python code then you must study the spaces rules defined as "PEP 8" at https://www.python.org/dev/pep... [python.org]
If you don't like the spaces rules then you can refuse to learn python and to start looking for another scripting languages.
And apparently, if you don't like the rules of English, you can post your prose as an AC on Slashdot.
Re: (Score:2)
Likewise, if you are unable to comprehend the main point of a simple sentence, you demonstrate that to everyone by posting as an AC on Slashdot.
Re: (Score:1)
Re: (Score:2)
Your effort was misdirected. I doubt that getting a pseudonym will help you with your grammar skills.
Not true (Score:2)
If you want to share code with the Python community, it will not be graciously received unless you follow the spaces rule, but if you want to write for your own use, or the use of your group, then many variations are acceptable. I, personally, always begin lines with tabs for indentation. If I want to share the code I run it through a conversion routine. But I prefer a 3 space indentation size, except when the nesting starts getting to deep, at which point I shift to a 2 space indentation. This is a lot
Can we quit with the myth that Python is slow ? (Score:5, Funny)
Python programs that I wrote 15 years ago are still running in production.
The "Java rewrite" that my manager wanted to do never got done.
In fact, a lot of the production Java code that existed back then couldn't be maintained and got reimplemented in Python.
Since the new servers are ~20 times faster, speed never really mattered anyway.
Python is full of "free" optimizations that most newbys are not even aware of.
When you learn to do things that "Pythonic" way, it really does put the clunky Java hack-jobs to shame.
Here another take on it:
https://www.pythonforengineers... [pythonforengineers.com]
Re: (Score:2)
He didn't suggest that the Java wasn't maintained due to backwards compatibility issues. He just said it wasn't maintained or was reimplemented.
If it was like many other places, it was simply because it's a bigger pain to read and maintain Java [github.com], generally speaking.
Re: (Score:2)
He didn't claim it got slower or that it became incompatible, he said it couldn't be maintained.
Re: (Score:3)
But some people are trying to use scripting languages to do *REAL WORK*, not serving web pages, but say, number crunching....and then, the ability to profile and know when to call out to something more efficient does matter, no matter which fractal of bad design you're using for scripting.
Re: (Score:3)
What could be that unicorn real work number crunching application that couldn't be written in 1/10 of lines of code of python in some numpy with a pinch of DAAL or tensorflow? Why most AI research who is not know to be low on the number crunching departement is done in Python?
There is even Games developped in Python (Sims 4, Eve Online, Civilisation IV, Pirates of the Caribbean Online...).
In my career, I have written my share of Assembly and C code for performance reasons when a 486 DX 400 was the must have
Re: (Score:2)
I think that's his point.
Do you know, for example, how much time your code is spending in TensorFlow and how much it's spending in the Python loop you wrote that calls TensorFlow? Profiling is doubly important when you're using an interpreted language because very simple optimizations can often get much greater payback, more so than for compiled languages.
Re: (Score:2)
Python is fine to prototype real work. Then, when it works, you rewrite the parts that do the heavy lifting in C. As an added benefit, you can run both implementation in parallel for testing purposes. I have had pretty good success with this approach.
Re: (Score:2)
Re: (Score:2)
I will just take it in the spirit it was said. Thanks!
Re: (Score:2)
Have you tried Cython? Used properly, Cython makes that transition pretty seamless. You start with Python code, which gets converted to C and compiled, but then you can add more and more C-isms to help out the translator, until it runs as fast as you desire.
Re: (Score:2)
You fucking C posers. Just use assembly. Stop reinventing wheels and get real work done.
Re: (Score:2)
Clueless statement is clueless.
Re: (Score:2)
I'll assume you're replying to the AC grandparent....
Re: (Score:2)
Yep, sorry. I must have slipped.
Re: (Score:2)
I've tried Cython a couple of times but it always came out slower than a real rewrite. I can't remember how much but enough not to bother.
I've mostly used boost python which is not the most popular option. It simplifies the wrapping a lot and I can make sure the C++ code is independent of any Python dependencies (except for the interface of course).
As for profiling, I really don't see the problem the article author has. I use the built in profiler and if the hotspot is "a_python_function" I optimize the pyt
Re: (Score:2)
Oh, the article author is an ad for a commercial product. The built in profiler works quite well. There are decent GUIs available too.
Re: (Score:2)
That sounds stupid. I much rather write C directly. And it does deprive me of the diversifying aspect were I test pure Python against C.
Re: Can we quit with the myth that Python is slow (Score:2)
And this is the recommended way to do this.
For me, except for the pure number transformations or high frequency sensor readings/processing, I haven't felt it worth it to replace a lot of python code for the minor shift in the overall performance. For most applications, Python is more than just good enough.
Re: (Score:2)
I have some symbolic simulation code that runs about 1000x faster for some real-world cases with the core in C instead of in Python. It really depends on what you are doing. Of course, if it had been running fast enough in Python, I would just have left it there, but sometimes you just cannot wait a week. And the Python version also had a much larger memory footprint. It was still hugely beneficial to have the pure Python version, as I found a lot of bugs by simply running both in parallel on randomized in
Re: (Score:2)
> when it works, you rewrite the parts that do the heavy lifting in C.
When it works you get onto the next project knowing that the next server upgrade will be far cheaper than the time you waste recoding it to C.
Spoken line an inexperienced amateur. Last time I did this, I got a speed-up of 1000x for real-world cases. No server-upgrade can reach or top that. It really depends on the task you are solving. Nobody would do this for simple business-logic, of course. But there are some people that implement actual algorithms and apply them to real-world data, you know.
Re:Can we quit with the myth that Python is slow ? (Score:5, Funny)
If you'd written them in C they'd have finished by now.
Re: (Score:3)
It's not a myth. Python is slow. You may choose to take that as a trade-off and that's fine.
Re: (Score:2)
For the intended target applications, it is also not a problem that Python is slow. You just re-implement the critical parts in C after the Python prototype works. Of course, this is something people that cannot code well in C cannot do. But that was/is the same problem with Java: It is pretty slow too. The idea there is as well to do the heavy lifting via JNI and C so that execution time is mostly spent in libraries. But as it turns out, most Java coders cannot do that, like, at all.
In the end, it comes do
Re: (Score:2)
In fact, a lot of the production Java code that existed back then couldn't be maintained and got reimplemented in Python.
Do you have an example that shows you actually know how to write Java code? Because if you can't even write Java in a maintainable way, something is wrong with you.....
Re: (Score:2)
I take it you have never done code-review fora larger project in Java? Because industrial Java code is routinely the most unreadable and unmaintainable trash you can run into. It is like the coders are trying hard to beat a Perl-coder with a bad attitude, bit at the same time have no skills at writing code at all. Incompetent coders produce unmaintainable code in any language and Java for sure has a very large share of incompetent coders due to its popularity. Anybody and their dog thinks they can learn to
Re: (Score:2)
I take it you have never done code-review fora larger project in Java? Because industrial Java code is routinely the most unreadable and unmaintainable trash you can run into.
I have. I agree it can get bad, but the code is still something that can be worked with, as miserable as it may seem.
Now imagine those same programmers were using C++. Without discipline, there would be memory corruption all over the place. Bugs would be nearly impossible, as the system would crash mysteriously all over the place. Java saves them from this mess.
What about Python or Javascript? Objects getting passed around all over the place with no type, or whose type changes over the lifecycle of the
Re: (Score:2)
I take it you have never done code-review fora larger project in Java? Because industrial Java code is routinely the most unreadable and unmaintainable trash you can run into.
I have. I agree it can get bad, but the code is still something that can be worked with, as miserable as it may seem.
Now imagine those same programmers were using C++.
They will not. These people will not get any real code in C++ to run.
Without discipline, there would be memory corruption all over the place. Bugs would be nearly impossible, as the system would crash mysteriously all over the place. Java saves them from this mess.
I disagree. Java allows them to produce something that seems to work, but causes more problems than it solves. These people have negative productivity.
"Memory corruption" is not the problem, it is just a symptom. Fixing the symptom does not fix the problem.
What about Python or Javascript? Objects getting passed around all over the place with no type, or whose type changes over the lifecycle of the object. No way of knowing what needs to be passed into a function (safer just rewriting it). No, these programs are unusable. At least in Java, when a function requires a string, you know you should pass a string. There may be other problems, but at least there is clarity on that point.
And the same is true for static type safety. It is a crutch. Those that need it cause numerous problems even with it, the problems are just in other areas. Technology cannot fix incompe
Re: (Score:2)
I disagree. Java allows them to produce something that seems to work, but causes more problems than it solves. These people have negative productivity. "Memory corruption" is not the problem, it is just a symptom. Fixing the symptom does not fix the problem.
Alright, examples, please.
And the same is true for static type safety. It is a crutch. Those that need it cause numerous problems even with it, the problems are just in other areas. Technology cannot fix incompetence.
Counter-example: static typing can reduce your bugs by 15% [ucl.ac.uk]
And that would be a benefit, because if they had to work in other languages, they would simply get fired for incompetence and be replaced with people that had a clue what they are doing.
No. There aren't enough good programmers in the world. Managers would rather have something that works poorly than doesn't work at all.
Re: (Score:2)
Counter-example: static typing can reduce your bugs by 15% [ucl.ac.uk]
It also decreases your productivity by 50%, which is most commonly spent between writing class definitions, untangling inheritance trees and maintaining conversion functions between functionally identical but differently named types.
Re: (Score:2)
It also decreases your productivity by 50%, which is most commonly spent between writing class definitions, untangling inheritance trees and maintaining conversion functions between functionally identical but differently named types.
Heh, now you're making up numbers. I can already guess that your bug tracker is filled with bugs.
Re: (Score:2)
Heh, now you're making up numbers.
2x is about how much faster Python interviewees code up the answers my questions, compared to C++ folks. And yes, it's an apples to apples comparison. I can tell they're roughly of equal intelligence by how long it takes them to figure out the algorithm.
I can already guess that your bug tracker is filled with bugs.
You're actually right. I have ~200 bugs assigned to me right now...
...for a C++ project I took over from supposedly very experienced C++ developers.
The Python one I previously worked on had ~30.
Re: (Score:2)
Re: (Score:2)
Maybe you can tell me how not to get a pile of bugs reassigned to you from a project you didn't know you were going to take on.
But in any case, I don't need an internet stranger to validate my capabilities as a software developer.
Re: (Score:2)
Is the bug count going up or down?
Re: (Score:2)
Mostly steady. We keep discovering new "features", such as the one that reboots the whole server if a user loads a particular page.
It only took a day to debug and fix that one. Turns out even things written in C++ can run out of memory and the previous guys didn't think the table can have so many rows.
On the plus side, at least we're getting a lot of new users.
Re: (Score:2)
Mostly steady.
Great! Hopefully soon it turns the corner and starts decreasing.
On the plus side, at least we're getting a lot of new users.
Great! Best of luck to you.
Re: (Score:2)
I've dealt with code like this recently. It's also why those environments using those tools could not upgrade to Python 3 until I replaced those critical business applications with shell scripts. They were quite simple scripts. They'd been written in Python because that was the mandated scripting language in that environment. I was fortunately not compelled to follow their coding standards, and discarding Python for the project made it much more staable.
Is it (Score:3)
Is it that slow? pypy seems pretty quick to me. Do programs start up that frequently these days outside of util scripts? Even then you can follow xargs lead and do more per execution.
Finally! Devel::NYTProf equiv? (Score:3)
After learning the differences in Python (and learning that I'd need to learn both v2 and v3), I started hunting for some of the tools that I use for Perl, like a profiler.
I couldn't find anything that could touch Devel::NYTProf. (Demo of that here) [youtube.com] Hopefully this can??
For one thing ... (Score:1)
... profiling is chickenshit and forging arms got Defense Distributed in a lot of twubba.
As any scripting language... (Score:2)
... and any language running in a non-jit virtual machine, Python is as fast as the library during library calls. It gets pretty slow when you code logic and calculations from elementary steps directly in Python. This is not new or surprising in any way.
Performs better than PowerShell (Score:3)
I replace ALL my PowerShell scripts on my widows servers with Python ones. Why it performed 10 times better!
Python also has fast statistical profiling (Score:1)
Statistical profiling lets you tune the overhead vs accuracy. By turning the overhead down low enough (e.g., by not sampling very frequently), you can be profiling your production code all the time and get very accurate data about what parts of your program are slow in real use cases.