Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Scikit-build-core driven wheel creation #1454

Merged
merged 8 commits into from
May 7, 2024

Conversation

DownerCase
Copy link
Contributor

@DownerCase DownerCase commented Mar 12, 2024

Description

Python wheels are now built via scikit-build-core! 🎉
This work is based off of Kerstin's attempt a few months ago.

You should have write permissions to this branch if you wish to make modifications 😄

Changes of this PR:

Removed

Updated

  • Runtime dependencies (ecal_core, hdf5) copied at install time (not build time) via CMake's install command
  • Wheels are built and uploaded by cibuildwheel
  • CMake 3.18 or newer is required if building the Python modules.
    • Access to the Development.Module component is strictly required for this workflow.
    • This is only enforced if BUILD_PY_BINDING is on.
  • The "wheel" CMake preset has been renamed to "python" and updated to reflect how wheels are now built (i.e: static where possible)
  • The eCAL version may be provided by scikit-build-core via the SKBUILD_PROJECT_VERSION CMake variable, this is the only change to the CMake scripts that interacts with scikit-build-core.
  • protobuf >=3.8 but < 4.X is listed as dependency.
    • By not shipping generated code, greater compatibility is possible. Should be able to extend to the python protobuf packages v4 and v5?

Added

  • The wheels now contain an autogenerated _version.py file containing the package version information
  • CMake has a new python component which has to be explicitly listed to CPack to be installed.

Identified issues:

Related issues

Related but may not close:

@DownerCase
Copy link
Contributor Author

DownerCase commented Mar 12, 2024

Ok. I'm being silly, I literally wrote that this won't work here due to the "v6.0.0-nightly" tag.

packaging.version.InvalidVersion: Invalid version: 'v6.0.0-nightly'

If you kindly push a tag called "v6.0.0.dev0" and rerun the wheels action everything should 🤞 be good.
Semver tags like "v6.0.0-alpha" or "v6.0.0-rc.1" also work as expected.

@KerstinKeller
Copy link
Contributor

KerstinKeller commented Mar 13, 2024

Hi, thanks so much for your PR, especially the great documentation of all issues.
I really tried gettings things to work myself for a day or two, but I ran into so many issues, that I moved on to other topics.

For me it's ok, not to ship the compiled protos.

We will try to retag with an alpha tag. But to my understanding, shouldn't 'v6.0.0-nightly' still be semver compliant?

@DownerCase
Copy link
Contributor Author

Hi, thanks so much for your PR, especially the great documentation of all issues. I really tried gettings things to work myself for a day or two, but I ran into so many issues, that I moved on to other topics.

Yup, this took a lot longer than I expected too, but worth it.

We will try to retag with an alpha tag. But to my understanding, shouldn't 'v6.0.0-nightly' still be semver compliant?

Yes, but its not Python version specification compliant.
setuptools-scm uses this regex by default to match PEP440 style versions so for Python its strictly enforced for the version to meet this format ([N!]N(.N)*[{a|b|rc}N][.postN][.devN]). There is another section further down that talks about schemes for developmental release (eg: X.Y.devN)

@DownerCase
Copy link
Contributor Author

@KerstinKeller Do you need anything further from me here? Everything is passing green now with the new tag (thank you!) except the cherry picker due to the missing label. If some pytests would lighten the validation load for you all and help get this merged I can do that, no problem.

@KerstinKeller
Copy link
Contributor

Sorry I was really busy last week. The PR looks already really good. We wanted to do some lokal tests before merging.
Pytests would totally make sense (we can also help write some), I guess they can also be executed as part of the wheel building process?
Anyways, we really appreciate your contribution, and we will likely merge it later this weeks.

@KerstinKeller
Copy link
Contributor

One more quick question: is there a reason that the ubuntu builds are done on Ubuntu 20.04 which is soon going to be deprecated. I think cibuildwheel will execute the actual build in a docker container? Or did I get something wrong here?

@DownerCase
Copy link
Contributor Author

One more quick question: is there a reason that the ubuntu builds are done on Ubuntu 20.04 which is soon going to be deprecated. I think cibuildwheel will execute the actual build in a docker container? Or did I get something wrong here?

Correct. cibuildwheel does all build in manylinux docker containers. https://cibuildwheel.pypa.io/en/stable/#how-it-works
Those platforms are chosen because I didn't change them when importing what you had previously done.

Will update to ubuntu-latest and windows-latest as it shouldn't affect anything 😄

@DownerCase
Copy link
Contributor Author

Sorry I was really busy last week. The PR looks already really good. We wanted to do some lokal tests before merging. Pytests would totally make sense (we can also help write some), I guess they can also be executed as part of the wheel building process? Anyways, we really appreciate your contribution, and we will likely merge it later this weeks.

No worries. Yes, the tests would run as part of cibuildwheel for every wheel

@FlorianReimold
Copy link
Member

I tested the whl files for Windows Python 3.9 and Ubuntu 22.04 Python 3.10 and everything worked out of the box. I tested the HelloWorld and HelloWorld Protobuf sample from the documentation. The systems were clean and didn't have eCAL installed.

Copy link
Contributor

@KerstinKeller KerstinKeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added two remarks, but I think they should prevent us from merging the PR.

@@ -159,7 +158,18 @@ option(CPACK_PACK_WITH_INNOSETUP "Create Innosetup installer for t

set(ECAL_INSTALL_PYTHON_DIR "bin" CACHE PATH "Location to install the Python extension modules. Might be set by setupdtools install.")

set(ECAL_BUILD_VERSION "0.0.0" CACHE STRING "Inject a build version if not building from a git repository")
if(DEFINED SKBUILD_PROJECT_VERSION)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a 100% friend of this, i'd prefer if the build could just inject ECAL_BUILD_VERSION from the outside.
But this may be cleaned up later on

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I could tell, this is the only way to get the package version from setuptools-scm into CMake, let scikit-core handle it.

lang/python/CMakeLists.txt Show resolved Hide resolved
@FlorianReimold
Copy link
Member

Hi @DownerCase,

I tried to build the python wheel locally on my PC and realized that I have no idea how to do that. Is there a guide somewhere how that can be done?

@DownerCase
Copy link
Contributor Author

There are a few methods but I do

python -m pip install build
python -m build

cibuildwheels can also be run locally if you desire.

https://packaging.python.org/en/latest/tutorials/packaging-projects/#generating-distribution-archives

@FlorianReimold FlorianReimold added the cherry-pick-to-NONE Don't cherry-pick these changes label Apr 3, 2024
@FlorianReimold
Copy link
Member

FlorianReimold commented Apr 3, 2024

Thanks, works like a charm!

Currently, we have the following python packages:

  • .whl files for Ubuntu amd64 and Windows amd64
  • .deb files for Ubuntu amd64 / x86 / arm64 / armhf

I checked whether I can reuse the old scripts to generate the .deb packages and that did not work (as expected). While I wouldn't really mind dropping the amd64 .deb installers as those users can switch to the manylinux pypi packages, I would still like to offer the arm64 users a way to install the python binding.

So, do you by any chance know how to
a) build arm64 whl files with the GH Actions or
b) build .deb packages for this python binding?

In the past I used the following script for generating the deb package:

override_dh_auto_install-arch:
	dh_auto_install --arch

	# Python install.
	set -e; \
	cd _build/python && for python in $(shell py3versions -r); do \
		$$python setup.py install \
			--install-layout=deb --no-compile \
			--root=$(CURDIR)/debian/python3-ecal5; \
	done
	find $(CURDIR)/debian/python3-ecal5 -name 'ecal-*-nspkg.pth' -delete

@DownerCase
Copy link
Contributor Author

a) build arm64 whl files with the GH Actions

Yes! cibuildwheel can do it: docs. I'll set that up here too

b) build .deb packages for this python binding?

That is outside of my knowledge. Theoretically that method could work on new enough versions of pip to support the build backend standard. You may need to add the minimal stub setup.py.

@DownerCase
Copy link
Contributor Author

Yikes, building arm64 in emulation is really slow.

I had a quick look at your deb situation, doesn't look great. For a start you'd be needing to disable the RPATH/install shenanigans as you'd want to depend on the system library from the main eCAL apt package.
On top you have to figure out how to get Ubuntu 20.04 to use new tools. The system packages on there have no idea about pyproject.toml as the ecosystem was stuck on setuptools and setup.py as the standard for so long... Doesn't seem impossible, just a lot of extra work... (Will be easier in the future when the debian packaging tools have fully caught up)

@DownerCase
Copy link
Contributor Author

Hmm. Over an hour to get all green CI ticks is unacceptable. Will rework to only trigger the emulated compilation on releases? IMO, I only need to beat the 37m windows build for daily PR workflows.

@FlorianReimold
Copy link
Member

Is there a way to have them trigger on releases and on "manual trigger" or something similar? I was also thinking about the same thing.

@DownerCase
Copy link
Contributor Author

Is there a way to have them trigger on releases and on "manual trigger" or something similar? I was also thinking about the same thing.

Theoretically I've done just that now 😄 It's set to only do the arm64 wheels on (pre-)releases being published or when the workflow is manually started.
See here for how to manually trigger a workflow: https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow

@FlorianReimold FlorianReimold merged commit e78be41 into eclipse-ecal:master May 7, 2024
11 checks passed
@DownerCase DownerCase deleted the feature/pypi_wheels branch May 7, 2024 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick-to-NONE Don't cherry-pick these changes
Projects
None yet
3 participants