Python's Steering Council Plans to Make Its 'Global Interpreter Lock' Optional (python.org) 21
Python's Global Interpreter Lock "allows only one thread to hold the control of the Python interpreter," according to the tutorial site Real Python. (They add, "it can be a performance bottleneck in CPU-bound and multi-threaded code.")
Friday the Python Steering Council "announced its intent to accept PEP 703 (Making the Global Interpreter Lock Optional in CPython), with initial support possibly showing up in the 3.13 release," reports LWN.net.
From the Steering Council's announcement: It's clear that the overall sentiment is positive, both for the general idea and for PEP 703 specifically. The Steering Council is also largely positive on both. We intend to accept PEP 703, although we're still working on the acceptance details...
Our base assumptions are:
- Long-term (probably 5+ years), the no-GIL build should be the only build. We do not want to create a permanent split between with-GIL and no-GIL builds (and extension modules).
- We want to be very careful with backward compatibility. We do not want another Python 3 situation, so any changes in third-party code needed to accommodate no-GIL builds should just work in with-GIL builds (although backward compatibility with older Python versions will still need to be addressed). This is not Python 4. We are still considering the requirements we want to place on ABI compatibility and other details for the two builds and the effect on backward compatibility.
- Before we commit to switching entirely to the no-GIL build, we need to see community support for it. We can't just flip the default and expect the community to figure out what work they need to do to support it. We, the core devs, need to gain experience with the new build mode and all it entails. We will probably need to figure out new C APIs and Python APIs as we sort out thread safety in existing code. We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable.
- We want to be able to change our mind if it turns out, any time before we make no-GIL the default, that it's just going to be too disruptive for too little gain. Such a decision could mean rolling back all of the work, so until we're certain we want to make no-GIL the default, code specific to no-GIL should be somewhat identifiable.
The current plan is to "add the no-GIL build as an experimental build mode, presumably in 3.13... [A]fter we have confidence that there is enough community support to make production use of no-GIL viable, we make the no-GIL build supported but not the default (yet), and set a target date/Python version for making it the default... We expect this to take at least a year or two, possibly more."
"Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility)... We think it may take as much as five years to get to this stage."
Friday the Python Steering Council "announced its intent to accept PEP 703 (Making the Global Interpreter Lock Optional in CPython), with initial support possibly showing up in the 3.13 release," reports LWN.net.
From the Steering Council's announcement: It's clear that the overall sentiment is positive, both for the general idea and for PEP 703 specifically. The Steering Council is also largely positive on both. We intend to accept PEP 703, although we're still working on the acceptance details...
Our base assumptions are:
- Long-term (probably 5+ years), the no-GIL build should be the only build. We do not want to create a permanent split between with-GIL and no-GIL builds (and extension modules).
- We want to be very careful with backward compatibility. We do not want another Python 3 situation, so any changes in third-party code needed to accommodate no-GIL builds should just work in with-GIL builds (although backward compatibility with older Python versions will still need to be addressed). This is not Python 4. We are still considering the requirements we want to place on ABI compatibility and other details for the two builds and the effect on backward compatibility.
- Before we commit to switching entirely to the no-GIL build, we need to see community support for it. We can't just flip the default and expect the community to figure out what work they need to do to support it. We, the core devs, need to gain experience with the new build mode and all it entails. We will probably need to figure out new C APIs and Python APIs as we sort out thread safety in existing code. We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable.
- We want to be able to change our mind if it turns out, any time before we make no-GIL the default, that it's just going to be too disruptive for too little gain. Such a decision could mean rolling back all of the work, so until we're certain we want to make no-GIL the default, code specific to no-GIL should be somewhat identifiable.
The current plan is to "add the no-GIL build as an experimental build mode, presumably in 3.13... [A]fter we have confidence that there is enough community support to make production use of no-GIL viable, we make the no-GIL build supported but not the default (yet), and set a target date/Python version for making it the default... We expect this to take at least a year or two, possibly more."
"Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility)... We think it may take as much as five years to get to this stage."
Re: (Score:2)
There's a lot of truth in what you say, although the multiprocessing module has long made it relatively easy to use forked Python processes with lightweight IPC needs. (Shared memory is hard with it )
But my immediate reaction to this was "Great! Now ALL my cores can spend a third of their time in malloc() and free() instead of just one core!" Idiomatic native Python code churns through objects like crazy....
Re: (Score:3, Interesting)
LOL at forked processes for "lightweight" needs. Threads were created because forked processes were NOT good for "lightweight" needs.
It's hysterical that Python thinks backward compatibility includes supporting bad threading code that might exist because their threading has been fake for its entire existence. A global interpreter lock is not part of the semantics of the language itself. Python never guaranteed thread safety without proper programming techniques, it's just that they realize that their mas
Re: (Score:3)
Python gain market share mostly by ease and speed of writing code that "works". It is well known that one use Python because they want to trade software running speed with software development speed.
Idiot-proof development environment by definition attract "idiots", or code written without "proper" techniques or by "shitty" programmers. What do you expect?
Re: (Score:3)
Whether real multi-threading will make this even worse is unclear, but I bet it does.
Re:It's 2023 and this is considered progress? (Score:4, Insightful)
Idiot-proof development environment by definition attract "idiots", or code written without "proper" techniques or by "shitty" programmers.
Python attracts smart people who are experts in complicated subjects like physics and biology, but not software engineering. Python's goal has always been to be a good language for getting things done, without people needing to become expert software engineers. That doesn't mean they aren't smart.
Re: (Score:2)
This is why Python should remain simple and limited. Everybody tries to do everything for all use cases and it turns into a complex mess which fuels the next more simple language. When will we just get a simple static language for non-programmers that can't do much more than the users need. Then make it loosely coupled with REAL languages for professional programmers to implement the hard stuff.
We keep repeating all our historical problems eventually. It might start out an improvement and avoid pitfalls a
Re: (Score:1)
I'm sorry you can't read, dude. They have lessons for that. I said lightweight needs for IPC, not a need for low-overhead IPC.
And the GIL isn't okay of the language spec. It's part of the runtime implementation. There are a bunch of internal data structures where mutual exclusion is provided by the GIL.
Oh plans, I see (Score:2)
They've been planning this for years now. More like the Python Sleeping Council, am I right?
Re: (Score:2)
Well, they've had to fight it out with the Steering Council for Python. Damn splittists!
The proposal is worth a read (Score:5, Interesting)
Lots of people have been asking for this for many years, but it's a very complex problem. For anyone who wants to know why this has taken so long or how they plan to fix this, I recommend reading the proposal [python.org] that was just accepted. It goes into great detail about the motivation, the obstacles and the proposed solutions.
Re: The proposal is worth a read (Score:2)
Re: (Score:2)
Stating that the GIL allows only a single thread to hold the python interpreter, and it can be a performance bottleneck is really stretching the limits of the english language. Complaint is valid.
Yes, it's a hard problem, but it is a way more important problem than most of the bullshit they seem to spend their time with. After being forced to port perfectly good python 2.7 code to python 3 for no really strong reason except that python2 is being forcibly deprecated, I have to wonder why, if they're going to
Re: (Score:2)
Some code will very likely break. You can't really evolve a language used by tens of millions of people without some risk of breakage.
But as painful as the 2 -> 3 transition was, it was quite necessary.
For some use cases, the option to disable the GIL (with eyes WIDE OPEN to the potential for things breaking) does solve a huge problem, and one they care a great deal about. Not me personally, as I don't use Python in places where top performance matters. But many, many others.
--disable-gil is absolutely
Re: (Score:3)
The author mentions a lot of ML use-cases and waffles a bit about other issues. Fine, however: ML is best done on a GPU or an FPGA (which can have thousands of threads) not so much on a multicore architecture (which only has tens of threads in comparison).
If you're doing ML without a GPU (other than a toy problem) then you're doing it wrong. The job of Python is to lay out the calculations and summarize the results afterwar
Re: The proposal is worth a read (Score:1)
Re: (Score:2)
It sounds like they put a lot of effort into thinking this through and creating a Python that will be better for at least most users than it is today.
Yet, my gut feeling says this change will create more problems, at least initially, than it solves. A lot of user code, not just internal Python code, relies directly or indirectly on the GIL. And probably a great number of libraries. My concern is that this could effectively split Python into two languages that are no longer compatible, and evolve separate
This will help my code a lot (Score:2)
I write engineering and scientific software and due to dealing with modelers it is easier to use Python that most other languages. Multiprocessing works but there are many algorithms that it just doesn't work on very well due to the overhead of copying objects across process boundaries.
I am really looking forward to these changes. I can make the code faster and easier to understand.
Or ... (Score:2)
Use Julia and don't be bothered by the GIL.