Python Implementation Pyston Aims To Speed Up the Programming Language's Code for Web Applications (techrepublic.com) 55
An anonymous reader shares a report: When Kevin Modzelewski and his colleagues at Dropbox set out to create Pyston in 2014, they had a very simple objective: to lower the costs of running Python code on Dropbox's servers, by making the code itself faster. "We were growing exponentially, so our server cost was growing exponentially," Modzelewski tells TechRepublic. "If we could get Python running faster, we would spend less money running Python." The original cost reduction initiative at Dropbox snowballed into a bigger project for Modzelewski when the company moved away from Python in 2017 and cancelled the Pyston project. He had realized while working on the language that there was a strong demand for faster Python among the developer community, and while there were plenty of tools around for improving the performance in smaller applications, there were none designed for big, business logic-type applications such as Dropbox.
"There's a lot of tools out there for helping you run Python faster, but there weren't any that were a good fit for Dropbox's use case," says Modzelewski. "This was an area of the Python market where a lot of money was being spent, but not very many tools were being developed for helping. It was under served." Fast forward to today and Pyston is now in version 2.2, and has been open-sourced, with Modzelewski and fellow developer Marius Wachtler now leading the project as co-founders. The latest implementation promises a 30% performance improvement over Python 3.8.8, with a key benefit being that developers can simply drop their Python applications into Pyston and get going, without having to rewrite their code. It's also a "completely separate thing" to what Modzelewski and fellow developers built for Dropbox some seven years ago.
"There's a lot of tools out there for helping you run Python faster, but there weren't any that were a good fit for Dropbox's use case," says Modzelewski. "This was an area of the Python market where a lot of money was being spent, but not very many tools were being developed for helping. It was under served." Fast forward to today and Pyston is now in version 2.2, and has been open-sourced, with Modzelewski and fellow developer Marius Wachtler now leading the project as co-founders. The latest implementation promises a 30% performance improvement over Python 3.8.8, with a key benefit being that developers can simply drop their Python applications into Pyston and get going, without having to rewrite their code. It's also a "completely separate thing" to what Modzelewski and fellow developers built for Dropbox some seven years ago.
Re: (Score:2)
Pffft! Real coders use butterflies.
Re: (Score:2)
Since they are talking about "big applications", some kind of compiled language seems a better fit anyhow for reasons beyond just performance. Java or C# may be a better fit for big apps or shops.
I generally find the RDBMS ends up being the performance bottleneck, not the app code, unless you are doing something silly. Most mass "data chomping" should be happening on the RDBMS, not app code. If it's happening in the app code, you probably are underutilizing the RDBMS. (Yes, I know there are exceptions.)
But
Re: (Score:2)
Since they are talking about "big applications", some kind of compiled language seems a better fit anyhow for reasons beyond just performance. Java or C# may be a better fit for big apps or shops.
That's my main beef against languages like python and javascript. I don't understand how to even start thinking about doing proper software engineering on them. There is just too much dynamicity in the typing of the language to really enable me to have proper invariants proven.
I just don't understand how one can be anywhere near confident in the applications written in these languages...
Re: (Score:1)
I avoid JavaScript, but TypeScript is similar in spirit to Mypy.
Re: (Score:1)
Re: (Score:1)
But mypy is more than just a way of avoiding testing. It keeps you from having to dig around in your code's callers to find out what types it needs to work with.
That's not a problem in a small, single-developer project, but unrealistic for large, multideveloper projects.
Re: (Score:1)
I just don't understand how one can be anywhere near confident in the applications written in these languages.
Python has really really good unit testing available for it. The unit tests give you confident the application works.
Re: (Score:1)
It's time to market startup thing. See http://www.paulgraham.com/avg.html for a glimpse in the mentality.
Maintenance is for the next guy so not our problem.
Having said that there are are recent advances in python and javascript towards static typing.
Re: (Score:3)
If you're doing anything but just using code to display end results of SQL queries, then you don't know how to code in SQL.
Re: (Score:2)
Re: (Score:3)
RDMBS's are optimized to handling large amounts of data, compilers and interpreters not so much, unless you're using something like COBOL which straddles the line. The only thing I do in code, since I primarily work in MVC frameworks, is format numbers, bold column headers, and maybe sum up columns. All the transformations and other nitty gritty, particularly working aggregating queries, where I am at times working with tables with hundreds of thousands of records, happens on the RDBMS.
Re: (Score:2)
Re: (Score:3)
I probably overstated it a bit. Obviously, particularly at the record level, there can be a lot of code involved in sanity checks, typing and the like. But I have seen code where guys are literally loading thousands of records into an array and doing all the transformations and aggregate work in code, which is insane. I guess on modern hardware where you might have a 64gb server that can handle that much memory allocation and 16 cores managing pointer allocation and the like, sure it can work, but it sure i
Re: (Score:2)
RDMBS's are optimized to handling large amounts of data, compilers and interpreters not so much
They're optimized to handle large amounts of (1) relational data (2) transactionally. Once either of those conditions is violated then doing said operation in an RDBMS *may* be suboptimal (depending on circumstances).
Re: (Score:1)
Generally if the app side has to do a lot of chomping that the database can't deal with, something is out of kilter: somebody is likely doing something wrong. If it's say AI or image processing, that's another animal, but generally you want those done in a tightly compiled language. Maybe separate the image processing onto dedicated servers and use Python as the glue language between the image servers, DB, and UI.
Re: (Score:2)
but generally you want those done in a tightly compiled language
So you would seem to apparently agree that the part about
compilers and interpreters not so much
was not quite correct?
Re: (Score:1)
Let's revisit "RDMBS's are optimized to handling large amounts of data, compilers and interpreters not so much".
I don't think RDBMS would be so great at image processing where lots of pixel-level transformations are needed. I could be wrong, but I don't think RDBMS are the best tool for "indexing pixels". Such is usually best done in app RAM.
I suppose if the image is very very large, then it may make sense, but that's a very niche need, like maybe an astrophysics lab with a map of the Milky Way.
RDBMS are be
Re: (Score:1)
Some business logic fits nicely as SQL (such as stored procedures), or at least part of it. "It depends".
Re: (Score:1)
If the app code is just reformatting a relatively small amount of data for display, then code optimization shouldn't an issue. Unless maybe the ad/spam engines use all kinds of Jiu-Jitsu that I'm not familiar with, doing mostly intranet stuff.
Re: (Score:2)
Re: (Score:3)
Python is interpreted. Most BASIC interpreters stored the program in p-code and interpreted the p-code (just like python).
In fact, "compiling to machine code" as you call it, is just the same process. The compiler "converts" the source code into p-code which used to run on the computer hardware directly. This is no longer the case. The p-code is interpreted by the CPU hardware and its microcode -- it no longer runs on the hardware directly.
Re: (Score:1)
Also, read up on the terminology for Java. Even back before it had a "JIT Compiler", it was considered "Compiled". Granted, there's gcj, which is merely AOT compilation.
Re: (Score:2)
Well architected code will split 'data chomping' between the RDBMS and the application code. It needs to be opinionated and based on solid understanding of the tradeoffs to be performant.
Re: (Score:1)
Maybe this case isn't. Some want the machines to (somehow) compensate for shit-ware.
If the processing is mostly lookups and sorting, then the DB is probably the best place to do it. If it's parsing, then the app side is probably better. But if your system depends on a lot of parsing, something is probably F'd up.
Re: (Score:2)
"Big Applications" are in the eye of the beholder. Dropbox is not a "big application". It is something that any minimally competent programmer should be able to write, test, and deploy in an afternoon.
Also, "bigness" is meaningless because it does not convey any useful information. Does it mean has lots of lines of code? Does it mean it is complicated (but only 2 lines)? It conveys a "wattage" problem and is not particularly meaningful.
Or by "big application" do you mean a really tiny simple one-liner
Re: (Score:2)
Only if it's buggy, leaky, lacks logs, and doesn't scale well.
The things you mention tend to be related. If you have a lot of customers you tend to spend more resources fine-tuning features, meaning more code, for example. I do agree the devil's in the in the details per "big", but being thorough often trigger
Re: (Score:3)
Dropbox is not a "big application". It is something that any minimally competent programmer should be able to write, test, and deploy in an afternoon.
Unlikely. As you forget all the "secondary use cases" - for which YOU perhaps already have a ready solution, but I not.
* sign up
* verify email (that obviously needs a little bit of infrastructure)
* log in
* security - can only see and access what he owns
* Clients for minimum 5 platforms (good luck with that in one afternoon)
* is synchronozation required/wanted
*
Re: (Score:2)
Since they are talking about "big applications", some kind of compiled language seems a better fit anyhow for reasons beyond just performance. Java or C# may be a better fit for big apps or shops.
When projects in Python or Ruby or Javascript (or php for that matter) get big, programmers start looking for a way to add types to the language. Might as well just start with compile-time types as far as I'm concerned.
Re: (Score:2)
I love C++ but it is not the be all and end all of programming.
I needed to write a quite restful service to slice some data in various ways. The data come in complex json format. They need to be slides in non trivial ways, indexed, and reserved in json formats. I can write this in C++. But why bother? IO is always going to be bottleneck anyway. Performance of that system is non critical. There are more python developers out there.
Python was a better choice.
Re: (Score:2)
Re: (Score:2)
Real coders use C++, the rest who can't understand/figure it out just complain and lob attacks until they realize they love C++.
Real coders use Smalltalk. Those that can't understand it/figure it out invent C++ instead and pretend it's the same thing but faster. It's faster, but it ain't the same thing at all.
Re: (Score:1)
Weight Watchers. (Score:3)
Take out all the white space. ;-)
Re: Weight Watchers. (Score:1)
Alternatives (Score:3, Informative)
For now, I think I'll stick with a mix of Pypy3 (JIT'd, single-threaded, high performance), Cython (allows pretty freely intermixing Python and C datatypes for compatibility and performance), Micropython (which appears to thread well) and Numba (a JIT for your innermost CPU-loop in CPython).
BTW, Python is dandy for large, multi-developer projects, but for that it's important to use Mypy (PEP484) or at least Pylint (with a tailored pylintrc to turn off some of the unnecessary warnings, while keeping the important stuff).
Re:Alternatives (Score:4, Interesting)
The Type hints from PEP484 are frigging amazing. Folks, don't think of it as a chore. This stuff makes your job *easier* by making your IDE finally understand whats going on, a task thats very difficult without type hinting. But with the hints? PyCharms type completion is borderline magica.
Plus I'm pretty certain that eventually all the compile-python schemes will end up requiring them to compile performant sensible code.
No. (Score:2, Insightful)
Sophisticated "web applications" need to die. They will live on because fools are sold on the idea of "put it in the cloud" but as a curmudgeon, I never shall. I refuse to give in to rent seeking, spying, and other bullshit behavior that needs to be connected 100% of the time.
Re: (Score:3)
Re: (Score:2, Interesting)
"web Front Ends" are usually developed out of laziness. It costs money to pay someone to write a proper front-end, but a web-kiddie can whip up a javascript infested (and godawfully fucking slow) "web interface" in an afternoon. This is why web front ends to anything are usually abysmal hunk of shit.
Re: (Score:2)
there are plenty of web front ends that disagree.
And your reasoning is wrong anyway.
Bottom line take the right tool for the job. If you write "a proper frontend", how do you distribute it? HTLM/JS comes from the server, simple.
Re: (Score:1)
If the person who signs your check wants their data in the cloud, you have to put it in the cloud. It's not your choice. If you want your own start-up, then form your own. Otherwise, the boss is the boss. Work World 101.
Re: (Score:2)
I was not referring to work, I was talking about people who willfully use sophisticated "web applications". If businesses want shoot themselves in the foot, fine but don't then turn around and support their efforts. Doing so only propagates the sickness.
Solution to wrong problem (Score:2)
Re: (Score:2)
I think the most important for Python's future, more than runtime optimization, is HPy: a better API for Python: https://github.com/hpyproject/... [github.com]
Granted, it'll take a while to move most "C Extension Modules" to HPy, but it should allow runtime developers a great deal more latitude in creating faster runtimes.
Pyston (Score:1)
Pissed-on⦠Really?
Can't wait to try it (Score:2)
So pyston is only excruciatingly slow instead of abysmally slow. Sounds great.
Re: (Score:1)
The more important point is that Developer Time is much more expensive than Machine Time, so it makes more sense to use something you can code in rapidly and effectively.
I used to think assembly language was the bee's knees. I got over it. You likely should move on too.
Why bother? (Score:2)
None of this makes any sense.
Re: (Score:1)
They want it droppier and boxier?
I suspect snoopware running AI so they can serve you the right underwear ads.
The first implementation (Score:2)
The first implementation that people have already Pyston?