Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows #44804

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows #44804

wants to merge 12 commits into from

Conversation

lysnikolaou
Copy link
Contributor

@lysnikolaou lysnikolaou commented Nov 21, 2024 •
edited by github-actions bot
Loading

Rationale for this change

There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.

What changes are included in this PR?

  • Add jobs to build free-threaded wheels on Windows
  • Move VCPKG-related stuff to a reusable base job

Are these changes tested?

No tests required.

Are there any user-facing changes?

No.

@github-actions GitHub Actions
Copy link

⚠️ GitHub issue #44421 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions bot added the awaiting review Awaiting review label Nov 21, 2024
@lysnikolaou
Copy link
Contributor Author

If someone could trigger the new jobs, that'd be really helpful!

@raulcd
Copy link
Member

raulcd commented Nov 27, 2024

@github-actions crossbow submit -g wheel-windows-*

@github-actions GitHub Actions
Copy link 

Invalid group(s) {'wheel-windows-*'}. Must be one of {'wheel', 'cuda', 'example-cpp', 'linux', 'homebrew', 'fuzz', 'integration', 'cpp', 'example-python', 'nightly-release', 'verify-rc-source-linux', 'c-glib', 'example', 'python', 'nightly', 'conan', 'verify-rc-wheels', 'packaging', 'r', 'verify-rc-source', 'verify-rc', 'nightly-packaging', 'nightly-tests', 'vcpkg', 'java', 'linux-arm64', 'verify-rc-binaries', 'conda', 'test', 'verify-rc-source-macos', 'ruby', 'verify-rc-jars', 'linux-amd64'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/12047528593

@raulcd
Copy link
Member

raulcd commented Nov 27, 2024

@github-actions crossbow submit wheel-windows-*

@github-actions GitHub Actions
Copy link

Revision: 78d8f9e

Submitted crossbow builds: ursacomputing/crossbow @ actions-86ba33e477

Task Status
wheel-windows-cp310-cp310-amd64 GitHub Actions
wheel-windows-cp311-cp311-amd64 GitHub Actions
wheel-windows-cp312-cp312-amd64 GitHub Actions
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions
wheel-windows-cp39-cp39-amd64 GitHub Actions

raulcd
raulcd reviewed Nov 27, 2024
View reviewed changes
Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, I triggered the CI jobs

ci/docker/python-wheel-windows-test-vs2019.dockerfile Show resolved Hide resolved
@github-actions github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Nov 27, 2024
@lysnikolaou
Copy link
Contributor Author

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

@lysnikolaou
Copy link
Contributor Author

Also, the failure on 3.9 is that py cannot be found when running py -%PYTHON% setup.py bdist_wheel || exit /B 1, which seems odd.

@raulcd
Copy link
Member

raulcd commented Dec 2, 2024

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

I haven't tested and I am unsure if this works but maybe we can try this (https://stackoverflow.com/a/21912169):
py -3.13t -c "import pyarrow" || echo ERROR && exit /B 1 instead of py -3.13t -c "import pyarrow" || exit /B 1

@lysnikolaou
Copy link
Contributor Author

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

@pitrou
Copy link
Member

pitrou commented Dec 4, 2024

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

Hmm, that one should be fixed in CPython AFAIU: python/cpython#111650

@pitrou
Copy link
Member

pitrou commented Dec 4, 2024

Ah, scratch that, the rationale is actually here:
https://github.com/pypa/distutils/pull/310/files

@pitrou
Copy link
Member

pitrou commented Dec 4, 2024

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

Well, stderr should already be shown by default, so it's weird that nothing gets displayed.

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from 95a0a53 to 83a76e3 Compare December 18, 2024 17:31
@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

@github-actions crossbow submit wheel-windows-*

@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

Given setuptools does not seem in a hurry to merge the distutils fix, I've tried to push a manual workaround in this PR.

@github-actions GitHub Actions
Copy link

Revision: 83a76e3

Submitted crossbow builds: ursacomputing/crossbow @ actions-5bfc22ac30

Task Status
wheel-windows-cp310-cp310-amd64 GitHub Actions
wheel-windows-cp311-cp311-amd64 GitHub Actions
wheel-windows-cp312-cp312-amd64 GitHub Actions
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions
wheel-windows-cp39-cp39-amd64 GitHub Actions

@pitrou
Copy link
Member

pitrou commented Dec 18, 2024 •
edited
Loading

Well, building the free-threaded wheel failed with:

LINK : fatal error LNK1104: cannot open file 'python313t.lib' [C:\arrow\python\build\temp.win-amd64-cpython-313t\arrow_python.vcxproj]

Edit: I've checked: that file exists in a Python 3.13 Windows install.

@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

@github-actions crossbow submit wheel-windows-cp313*

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from 0d30b99 to 55e4cc0 Compare December 19, 2024 07:42
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions
Copy link

Revision: 55e4cc0

Submitted crossbow builds: ursacomputing/crossbow @ actions-1076ab812c

Task Status
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions

@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from e4c9baf to 184510c Compare December 19, 2024 10:20
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from 184510c to b5fbae1 Compare December 19, 2024 10:51
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from b5fbae1 to 771cf66 Compare December 19, 2024 13:10
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions
Copy link

Revision: da27ca9

Submitted crossbow builds: ursacomputing/crossbow @ actions-4bc1383e5a

Task Status
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions

@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

@github-actions GitHub Actions
Copy link

Revision: de2f14a

Submitted crossbow builds: ursacomputing/crossbow @ actions-bfacfef460

Task Status
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raulcd raulcd raulcd left review comments

@assignUser assignUser Awaiting requested review from assignUser assignUser is a code owner

@jonkeane jonkeane Awaiting requested review from jonkeane jonkeane is a code owner

@kou kou Awaiting requested review from kou kou is a code owner

Assignees
No one assigned
Labels
awaiting change review Awaiting change review Component: Python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lysnikolaou @raulcd @pitrou