ChatGPT-Powered 'Scalene' Offers Efficiency Suggestions for Python Programmers

The tech site IT Brew looks at an open-source tool that "uses AI to offer efficiency-minded suggestions to Python coders." Known as "Scalene," the profiler — a kind of debugger for performance issues — has been downloaded more than 900,000 times on GitHub. "It's awesome in general, and amazing for an academic project," UMass professor Emery Berger, who worked with PhD students Sam Stern and Juan Altmayer Pizzorno on the open-source tool, told IT Brew...

Scalene measures how much time and memory is spent on each line of code — both on average and at peak, [and] how much time is spent in efficient libraries and how much is spent in Python... By selecting a lightning-bolt icon, a user can "leverage the engine that powers ChatGPT to get an optimization" suggestion, Berger said. In one demo he showed IT Brew, an output recommended a less-memory-intensive move to reduce a very large array created by the code...

"If your Python code already runs fast enough, then you don't need a profiler. But if it's running slow, I think it's a very convenient profiler to reach for," Berger said.
Link via Dev News .

Given The

[Barny]

Given the target audience, how much do you want to bet most of those downloads were from a bad Python script that kept fetching it over and over?

Speed is not everything.

[devslash0]

These kind of speed-ups come very naturally to professionals proficient with Python (such as myself). However, what matters more than speed is readability. Trying to teach students and novice programmers programming focusing on speed is bound to make them produce ugly, unreadable and unstructured code later on in their development life.

import this:

Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Complex is better than complicated.

Flat is better than nested.

Sparse is better than dense.

Readability counts.

Special cases aren't special enough to break the rules.

Although practicality beats purity.

Errors should never pass silently.

Unless explicitly silenced.

In the face of ambiguity, refuse the temptation to guess.

There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch.

Now is better than never.

Although never is often better than *right* now.

If the implementation is hard to explain, it's a bad idea.

If the implementation is easy to explain, it may be a good idea.

Namespaces are one honking great idea -- let's do more of those!

Re:

[narcc]

The simplest way to improve both speed and readability is to stop using Python.

Re: Speed is not everything.

[devslash0]

Readability can be improved by stopping forcing typing everywhere. Speed? If you've come to Python for speed comparable with compiled languages then you're in the wrong place. Go educate yourself and choose a tool suited to your needs instead.

Re:

[Rockoon]

You never hear "use another language if you want speed" from someone that just wrote objectively good code.

You only hear it from people that just wrote objectively shit code.

Re:Speed is not everything.

[Mr. Dollar Ton]

Yep. Luckily, Perl 5.x is still available.

That is advanced bullshit

[gweihir]

With a scripting language, you do not do detail optimization with a profiler. It is the wrong tool for the wrong job. You do algorithm optimization, you may look at execution times of larger tasks, but that is essentially it. If you want to optimize runtime then you move the critical parts to C and _there_ doing profiling makes sense. And on C-level, you can also optimize memory footprint, which is often very important for performance and even do cache-awareness. Forget doing that on Python level.

And always remember, code optimization is for experienced experts only. If you are not one, you are far more likely to shoot yourself in the foot and may make your code unmaintainable or create even worse problems.

Re:

[ls671]

Indeed, python is usually != efficiency and optimized code

Re: That is advanced bullshit

[devslash0]

Python is not a scripting language. It is a general-purpose interpreted language. Yes, it is frequently used as a scripting language but it's just a tiny part of its use cases.

Re: That is advanced bullshit

[vivian]

Python is a great language for knocking out code fast and flexibly, but if performance is a concern, it's the wrong tool for the job.
I worked on a large ROS project that used python, C and C++ for various nodes to control a multi robot fab line. All the heavy lifting such as trajectory planning, evaluating laser scan data and updating world state was done in C++, with real-time controllers written in C, and a lot of task pipelining and messaging between various bits and pieces using Python. Occasionally when some new piece of equipment was being added to the system it might be controlled using a python script, but if it got to the point where that looked like it would be a bottleneck, it made a lot more sense to just rewrite or move some slow critical part out of the python script into a C++ node.

Re:

[gweihir]

It depends. I did some high-performance stuff with Python (simulation). The way you do it is make a prototype in Python, then port the heavy lifting to C. This also allows for nice testing of the C code whether it delivers the same results as the Python code on test cases. In my real world application scenario, I brought the runtime down from about a month to about an hour for a run.

Of course, you need to be firm enough in OO to understand how to do it in C, but it is really not that hard and the C interfac

Re:

[Mr. Barky]

Not all code needs to be maximally fast - just faster than it is at the moment.

Sometimes performance problems are algorithmic and moving it to C++ will only marginally help. A profiler is there to help figure out where problems are. It is up to the programmer to study the output and figure out what needs changing. Sometimes minor code changes can lead to large speed improvements. The advice on moving it to C++ code is potentially advice to increase the complexity of the project - two languages, two build sy

Re:

[gweihir]

Well, a scripting language is a type of "general purpose interpreted language". One typical characteristic is that you can use it interactively (check) and place the plain-text script in a text file with a "hash-bang" at the top (check). It is also not a systems-language which you can use, say, for compiled kernel modules or low-level manipulation or high-performance stuff (check on all three). And some more: https://en.wikipedia.org/wiki/... [wikipedia.org]

So, yes, Python is a scripting language. It is also a general purp

Re:

[Draeven]

Isn't that what the tool is proposing to do? It profiles your python code and finds opportunities (sections of code that run poorly) and makes suggestions for libraries or algorithms to improve it? In theory proposing libraries written in C, perhaps?

Re:

[narcc]

In addition to algorithmic complexity, I'll add data structures. Though I'm not confident that developers these days would even know where to begin. There are other things that can be done to improve performance without introducing needless complexity. Simply reducing needless memory allocations and deallocations, for example, can dramatically improve performance.

code optimization is for experienced experts only [...] make your code unmaintainable or create even worse problems.

I get what you mean, but there's a lot of ground between 'picking better algorithms' and whatever Mel [utah.edu] gets up to these days.

Optimization doesn

Re:

[Mr. Barky]

I am not sure why you (and many others) are so dismissive of this. Optimizations are often useful in any language where the program takes longer than a user would like (and the benefit or reprogramming is worth the effort). A profiler is just a tool to identify slow places in your code. Once you've identified the hotspot, you can then analyze its origin and figure out what to do about it. Maybe they're algorithmic (e.g. using an N^2 algorithm instead of N Log N), maybe they're just a poor implementation of

If you couldn't code, you sure will never learn it

[thesjaakspoiler]

with these kinds of tools.

I have an efficiency suggestion too.

[linuxgeek64]

Use a real language.

I remember CODE REVIEWS with live persons

[Canberra1]

Where your boss reviewed all your code, and was an expert coder/programmer. Then occasionally we all got together in one room, a projector screen and discussed the code that passed reviews - and compile options and testing outcomes. Then management came along , sacked the testing team for finding too many defects, and the code review team for discovering poor programmers that only knew cut and paste - but were very quick at it. (Spaghetti prototype code) They had to be fired for upsetting DVI hires and slave B visa imports. Then sacked the usability testers, and the DBA's who looked for table scans etc. Better to do nothing and go for production testing.

Re:

[Rockoon]

..when nobody would tell you that using bitwise and ("&") is "premature optimization" as it is "more complicated" than using remainder ("%")

The reason you didnt hear it is because that boss would fire any fuck that got confused by bitwise operators .. if you are confused by bitwise operators, there is the door, leave... dont come back... we want programmers that know the basics, not ones that are confounded by the basics.

Re: I remember CODE REVIEWS with live persons

[LindleyF]

If you are arguing over whether to use bitwise operations you may be bikeshedding on questions that don't actually matter. Write code that is first correct, then maintainable, and only then maybe if you need to, worry about hand-optimizing.

Re:

[Rockoon]

Word salad.

"Bikeshedding"

The bloke declaring "premature optimization" and making a stink of it about & vs % .. they are the ones "bikeshedding"

I cant help but notice that you picked a "side" and now you are claiming that the "other size" is guilty if what the "side" you picked did

Re: I remember CODE REVIEWS with live persons

[LindleyF]

I'm saying it usually isn't worth a debate. So long as the unit tests verify the code does the right thing, move on. Getting stuck on disagreements over implementation details shouldn't happen unless it's likely to affect readability or maintainability.

Re:

[Canberra1]

Unit Tests. What the F are those?! Programmers who write the code are expected to write their own unit tests, and are often ignorant of what or the range of values within the database. Classics are 'Court Orders' , Being Dead, with a lengthy contested probate, and companies shredding data/emails/evidence after seven years when the court case is still live. Married names, sex changes add to exceptions in the real world. Walkthrough's catch these exceptions. Hence the new phrase 'Production Testing' and 'no

Re:

[Entrope]

I would go so far as to say that they are universal. I cannot even think of a programming language that does not have all the same ones.

That says a lot more about you than about programming. Brainfuck [wikipedia.org] is a stupidly small Turing-complete language. Prolog [wikibooks.org] is a language with a fundamentally different approach than most imperative languages. Lisp exposes the program itself as a data structure, so a program can inspect and change itself at runtime.

Re:

[mistergrumpy]

Not that much of a bug. In your example, isLeapYear() should return undefined for any year < 1582, as the Gregorian calendar is undefined before that. Even the Julian calendar is only defined for years >= 45.

Re:

[angel'o'sphere]

Interesting.
I would never come to the idea to test for a leap year with year & 3. (And if I think about it now, it is wrong anyway. Anding with 3 leaves the last 2 bits. So it needs to be year & 3 != 0)

It does not make any sense.
You are masking out the higher bits. What the fuck has that to do with a leap year?
It makes everyone look at the code: oh, the function name is "isLeapYear" and he is just anding a random looking number to the integer year?

It is wrong anyway, 1900 was not a leap year ... nei

Re: I remember CODE REVIEWS with live persons

[madsh]

Calendars are hard. If you think you can implement one or improve one (while solving other problems) you are likely wrong.

Re: I remember CODE REVIEWS with live persons

[LindleyF]

The correct way to implement IsLeapYear() is: don't. Use a well-known civil time library instead. Abseil has one.

Re: I remember CODE REVIEWS with live persons

[cyber-vandal]

Ah yes the "good old days". I remember having to try to figure out all that 70s junk code for Y2K. Thanks for making all the consultancy money necessary grandpa.