Drop PyPy compatibility?

[jcrist]

During a call today the issue of dropping PyPy compatibility was brought up.

We currently loosely maintain PyPy compatibility (we don't test with PyPy at all), but some design/process decisions take PyPy into account. For example, currently the cython build of the scheduler requires that all annotations also work in pure python mode so it can run without cythonization. These annotations muddy the code, and don't work as well with type checkers like MyPy. Additionally, avoiding separating out the cython bits of our code makes it harder to understand what bits have been optimized for cython, and also make it harder to perform further c-extension level optimizations.

Since many users are using libraries that don't support PyPy, PyPy is likely mainly useful for speeding up the scheduler (leaving the workers running cpython). I don't know of any users running in this configuration though (most users use the same python environment for both scheduler and workers), and suspect that the maintenance burden outweighs the benefit.

As such, I suggest that we drop our (loose) PyPy compatibility requirement. This would result in no immediate changes to the codebase, but going forward we might make changes that break compatibility.

[jcrist]

Ope, pinging some relevant people from different orgs: @quasiben, @jakirkham, @jrbourbeau, @fjetter, @jsignell (apologies if this issue is relevant to you and I didn't ping you).

[jakirkham]

I think @mariusvniekerk has used PyPy support in the past.

Not sure who else. Random issues like this one ( #2734 ) pop-up from time to time suggesting people use this, but have the occasional minor problem.

[jakirkham]

cc @mattip (for awareness)

[jakirkham]

With Cython annotations, are these only there because of PyPy? At least my understanding originally when working on those was we cared about having pure Python with no Cython as a run option. This can matter outside of PyPy (for example some linters don't work on Cython). Additionally this adds a compilation step for users or packagers. IOW are we now comfortable requiring Cython? Should add this is somewhat separate from PyPy as Cython can also be used with PyPy.

I'm curious if you have some examples of libraries not supported on PyPy that are important. Asking this as conda-forge builds PyPy packages for all sorts of libraries.

Edit: Happy to discuss how the Cython parts can be improved further, but maybe this is best done in another issue.

[mattip]

I am missing a bit of background.

• It seems there is a python-only wheel on PyPI, https://pypi.org/project/distributed/2022.1.0/#files. Where does cython come into play?
• Are there benchmarks or indications that this library is a bottleneck in some workflow that means people might be searching for a speed-up? If so it might be nice to try PyPy. If not, then why would someone bother?

[fjetter]

It seems there is a python-only wheel on PyPI, https://pypi.org/project/distributed/2022.1.0/#files. Where does cython come into play?

We are indeed not publishing any cythonized builds (#4442). We attempted to implement an optional cythonization for the scheduler (

distributed/setup.py

Lines 26 to 30 in 337f152

	# To enable Cython, add to pip install one of the following:
	# --install-option="--with-cython"
	# --install-option="--with-cython=annotate"
	# --install-option="--with-cython=profile"
	# --install-option="--with-cython=annotate,profile"

) but we never published a build so this would require users to build it themselves. In fact, we are actually also considering to drop support for this, see #5685

Are there benchmarks or indications that this library is a bottleneck in some workflow that means people might be searching for a speed-up? If so it might be nice to try PyPy. If not, then why would someone bother?

There are a bunch of issues open about speeding up the scheduler. It's known to be bottlenecked for high task workflows (e.g. once you reach 1M tasks). There's been some extensive profiling ongoing as well. There are actually many tickets in our tracker about this topic. Here are just a few important ones

• Accelerate Scheduler with Cython, PyPy, or C #854
• Profiling Scheduler Performance #4443

[jakirkham]

Sorry for taking us down a tangent here, I'm doubtful that Cython and PyPy are incompatible. We aren't doing anything that is likely to cause PyPy significant issues (there may be performance differences though).

[mrocklin]

What is the status here?

DId we drop pypy support? Were we even maintaining this?

If we wanted to push forward on dropping this, what would be a next step (if there are any)?

[jrbourbeau]

DId we drop pypy support? Were we even maintaining this?

We decided to drop support at the monthly community meeting on 2022-02-03

If we wanted to push forward on dropping this, what would be a next step (if there are any)?

I think the actual code changes for dropping support are fairly minimal. See #6029 and dask/dask#8863

[jakirkham]

There may be packaging changes as well. Filed issue ( conda-forge/distributed-feedstock#201 ) to figure that out

[jcrist]

Resolved by #6029. Closing.

[mrocklin]

It was nice to see people moving quickly here

…

On Thu, Mar 31, 2022 at 1:52 AM Jim Crist-Harif ***@***.***> wrote: Closed #5681 <#5681>. — Reply to this email directly, view it on GitHub <#5681 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTF2XTFINGWXYNAMQ6TVCVDSHANCNFSM5MP35NSA> . You are receiving this because you commented.Message ID: ***@***.***>