[To owner] How do you manage updating Python dependencies

[ltsuda]

I'm very curious what are the steps, procedures that @mdmintz follow to update each dependency.

How do you know which dependencies have a new version? If you can upgrade them without breaking project (his might be covered by infra tests), what about child dependencies too because they are probably upgraded automatically when upgrading a top-level package.

[mdmintz]

I run this everyday:

pip list --outdated

It tells me which dependencies have an update. Then I try running tests with the latest versions. If successfully, the minimum version is bumped. If not successful, (or there is a conflict), the dependency stays where it is. As of today, the only dependency that can't go to the latest version is pytest-html (due to bugs in newer versions).

For trusted dependencies, I use >= so that people can refresh to a newer version on their own. For untrusted dependencies, like selenium, I use == to pin an exact version until tested and it's safe to upgrade. By "untrusted", I mean those packages have been known to make changes that break backwards compatibility. Here's an example of that: https://stackoverflow.com/a/72754667/7058266 (find_element_by_xpath("XPATH") was removed, and people had to switch to find_element("xpath", "XPATH")). This next one happened recently: https://stackoverflow.com/a/76432322/7058266 (selenium 4.10.0), where webdriver.Chrome() had most of its args moved or removed. Once I changed internals, SeleniumBase was able to upgrade those libraries safely. Note that SeleniumBase does the hard work for people: Tests never break backwards compatibility (I ran a test unmodified from 6 years ago, and it still worked today).

In some special cases, instead of using >= or == in the requirements.txt and setup.py files, a special range is used instead: Eg. urllib3>=1.26.16,<2.1.0, due to urllib3/urllib3#2168 (ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168). This range lets people use urllib3 1.26.16 to avoid the issue, while letting others use urllib3 2.0.3 if it works for them. I was actually doing some research on that one today, and I may force users of Python<3.10 to use only 1.26.16 right now, while letting users of Python>=3.10 have the current range (to prevent errors when not pinning urllib3).

Dependencies change often, so I run pip list --outdated often to check on them.

[ltsuda]

Wow! What a fast and awesome response 🥇

I really appreciate your answer and I found it very useful to me, so thank you!

I'm assuming and looked through setup.py that in install_requires are the exactly the same content as the requirements.txt, excluding the test packages that are in the extras_require field. Another thing I'm assuming is that you used pip freeze, at the beginning of this project, to get all of the dependencies regardless if some or most of them are dependency of dependency and follow up with the pip list --outdated every since.

Again, thank you for the pretty detailed answer :)

[mdmintz]

Yes, setup.py and requirements.txt are maintained together in sync. The pip freeze command was used at the beginning to generate the initial requirements. Updates occurred based on the results of pip list --outdated and follow-up testing.