Initializing AiopgConnector fails on Windows with Python 3.8

[t-eckert]

Not an issue with Procrastinate, but with the aiopg package, but there is a way to fix it in this package.

This issue gives more context on the problem as the tornado package has the same bug: tornadoweb/tornado#2751

In short, loop's add_reader() now "not implemented" by default on windows and asyncio platform uses it. This is probably related to this issue too: #126.

As a Windows user on Python 3.8, when I run the tutorial from the quickstart, I get:

Scheduling computation of 2 + 3
  File ".\tutorial.py", line 23, in <module>
    sum.defer(a=a, b=b)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\tasks.py", line 114, in defer
    return self.configure().defer(**task_kwargs)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\jobs.py", line 151, in defer
    id = self.job_store.defer_job(job=job)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\store.py", line 35, in defer_job
    result = self.connector.execute_query_one(
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\utils.py", line 121, in wrapper
    return sync_await(awaitable=awaitable)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\utils.py", line 171, in sync_await
    return loop.run_until_complete(awaitable)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1520.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\aiopg_connector.py", line 31, in wrapped
    return await coro(*args, **kwargs)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\aiopg_connector.py", line 67, in wrapped
    return await coro(*args, **kwargs)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\aiopg_connector.py", line 249, in execute_query_one_async
    pool = await self._get_pool()
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\aiopg_connector.py", line 220, in _get_pool
    self.set_pool(await self._create_pool(self._pool_args), _external=False)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\aiopg_connector.py", line 31, in wrapped
    return await coro(*args, **kwargs)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\procrastinate\aiopg_connector.py", line 201, in _create_pool
    return await aiopg.create_pool(**pool_args)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\pool.py", line 155, in from_pool_fill
    await self._fill_free_pool(False)
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\pool.py", line 193, in _fill_free_pool
    conn = await connect(
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\connection.py", line 38, in connect
    coro = Connection(
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\connection.py", line 92, in __init__
    self._loop.add_reader(self._fileno, self._ready, self._weakref)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1520.0_x64__qbz5n2kfra8p0\lib\asyncio\events.py", line 501, in add_reader
    raise NotImplementedError
NotImplementedError
Exception ignored in: <function Connection.__del__ at 0x000001903A3B6430>
Traceback (most recent call last):
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\connection.py", line 512, in __del__
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\connection.py", line 310, in close
  File "C:\Users\t_eck\.virtualenvs\Procrastinate-m37gheT2\lib\site-packages\aiopg\connection.py", line 285, in _close
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1520.0_x64__qbz5n2kfra8p0\lib\asyncio\events.py", line 504, in remove_reader
NotImplementedError:

What has worked for me is adding

import sys
import asyncio

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

To the head of my file. While aiopg sorts out this issue on their end, it may be worth adding the snippet to this package. But I could also see how that could be considered "monkey patching". If so, it may be worth noting the quirk in the installation section of the docs.

Thanks!

[ewjoachim]

Hello, thank you for your issue! I'm definitely interested in making Procrastinate available to a broader set of user! That being said, I don't have a Windows machine to test.

We can work on it on our end, but would you be interested in a code / docs contribution towards this goal? Our path to supporting Windows could be:

• Integrate all quick fixes that Windows users mentions, blindly for now
• In parallel, switch our CI to GitHub Actions. I've been meaning to do that for a while (we'll do than on our end but of course, you're welcome to contribute if you want)
• In GitHub actions, set up our CI so that it runs on windows too. That way, we'll know if it fails in the future.

There are things that might already be failing on Windows:

• All the code revolving around process signals. I'd say this is UNIX-based, so not sure if it works. We might have to change a thing or two.
• Does environment variables work the same way under Windows ? We might have to adjust some documentation
• I'm pretty sure the contributing doc only mentions Linux & MacOS, we might need a section on Windows.

So here we are :) We will definitely need a bit of help to kickstart Windows compat, and maybe someone to occasionnaly ping if the Windows build doesn't run.

[t-eckert]

Happy to help! If you want to assign issues to me, feel free!

[ewjoachim]

Well thanks! I've assigned you this one then. Feel free to give a go to your suggested changes, and if it makes procrastinate work for you, I'll be glad to merge. I'll create the other tickets :)

Note: let it be said explicitly: I appreciate your offer a lot, but if, for any reason, you prefer not to commit time to this for now, it's perfectly ok to say so, and it's even better if you can tell us, so that someone else can take the ticket :)

Have fun, good luck 👍

[ewjoachim]

I've created a tag "Linked to: Windows". This way, we'll be able to track tickets more easily if they need someone familiar with Windows environments to take care of

[t-eckert]

Just a note. I haven't forgotten about this. I discovered the bug while working on a client project. I wrapped up the project on Friday and am now on vacation. I intend to revisit this when I return next week.

[ewjoachim]

Nice :) It's always reassuring to have news, but it's perfectly ok for you to take all the time you need :) This is voluntary work, in the end.

Wishing you all the best for your vacations 🏄 🌴 and stay safe :)

[ewjoachim]

Just so you know: the CI is now running with GH Actions, so we can now check with Windows (and Mac). It's not setup yet but feel free :)

[TheSima]

So, you also can hotfix it in asyncio package replace default event loop by change line in windows_events.py
DefaultEventLoopPolicy = WindowsProactorEventLoopPolicy
to
DefaultEventLoopPolicy = WindowsSelectorEventLoopPolicy

[t-eckert]

Good note, I have tried to tackle this twice and gotten stymied on setup for Windows. I have also thought about writing a setup script for Windows as a yak shave.

[wlievens]

Is there any update on Windows support?

[ewjoachim]

@wlievens Would you be interested in giving it a go ?

[aleksandr-shtaub]

This issue seems interesting for me. I'll see what I can do.

[ewjoachim]

Good idea. I'll create an official issue for windows support, so this issue can be focused on the original problem