-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poetry doesn't (correctly) follow markers for non-PyPi Dependencies (at least w/ Pytorch) #9383
Comments
Also, don't know if there's a way to specify which package versioning is prioritized, but that would be exceptionally helpful in general. I know there is source-wise. But one of the annoyances that exists is Datasets doesn't recognize or has been set by Pytorch to look for a stable version of Torch. I don't care if Datasets is buggy/doesn't work as intended @ 2.19, whereas I need Torch @ 2.4. |
You did not provide It may be that you are trying to add |
Whoops, had included so much info, figured there's no way more would help heh. Attached it below. Ignorant of how Poetry works behind the scenes, the direct URL install attempts were "phishing" -- hoping that if Poetry latched on and installed from the links it would refresh the indexed file-SHAs it had on-file with the SHAs of links/files with X/Y/Z endings (all due to having a new approach/markers it would look for). And if the Pyproject changed, could always revert the PyProject.toml to its former repository-based state for Torch. |
Then this all is working as expected and should be closed as user error, thanks! |
Huh? I'm saying that was my own troubleshooting, the issue still exists/persists with or without that. Same goes for with/without my markers.
|
we did this already, you have asked poetry to install torch both as a direct url dependency and also from a specific repository. it cannot do both at once, and telling you so is not a bug. |
This is what happened doing |
I feel as though we are talking past each other, but I cannot tell what you are trying to report. You have provided a screenshot showing poetry behaving correctly, we have discussed how that behaviour is correct, I do not follow what the conversation is now about. Most torch issues are in some way duplicate #6409, perhaps you will find whatever it is you are looking for there. |
Poetry works correctly for 1 of the 2 methods so I confirmed that for you (the URL dependency). What is not working as intended (despite attempts to reel in Poetry's problemed behavior with markers) was finding the befitting version for my platform. Though I may have just figured it out. I supplied the URL that Pytorch.org suggests for using as their index which is https://download.pytorch.org/whl/nightly/cpu and this is what the poetry.lock has provided as options for the package.
As I've mentioned, it opts for "2.4.0.dev20240505+cpu" -- even though that is not an option for Macs. Even if I put "^2.4.0.dev20240505" it'll do +cpu only. I use the /cpu/ index because that's what Pytorch.org suggests and it is smaller list to look through than https://download.pytorch.org/whl/nightly/ (and to clarify /cpu/ index still has torch-2.4.0.dev20240501-cp39-none-macosx_11_0_arm64.whl, etc., etc. I have a feeling that without the /cpu/ in the URL, it will work as intended...just really, really slowly. But perhaps my markers will work now. We will see when I try this with my other Poetry project. Decided to try sans /cpu/ without markers first for simplicity/pure testing which is taking forever. Sadly nope. Now it is going for roc-m ones. I wish there was a whl/nightly/mac. AFAIK that doesn't exist. For now I will stick to the URL dependency method and I'll use a script to trigger when I update Poetry dependencies, it will take care of changing the "20240505" part of the URL to keep up with the nightly updates unless there's a way to do that with Poetry directly in the TOML I'm unaware of. |
Then this is indeed all known and duplicate, and all on torch:
The good news is that per those issues torch seem willing to engage Anyway there is nothing for poetry to do here, please close |
Description
So I have been using Poetry for a couple of weeks and its still an improvement over Pipenv by a longshot but there seems to be an underlying issue (and perhaps it is actually entirely on Pytorch's end/their bad metadata, rather than some foundational Pip wheel downloader problem.
I am using a Macbook Pro M1. And these are the markers I'd like to use -- but they are never followed, particularly on Pytorch.org downloads. I've had better luck with Tensorflow, for example, but I only need 1 or 2 of their packages direct-from-source as opposed to the nigthlys avail. through Pytorch.org so I can't say for sure.
markers = "platform_machine == 'arm64' and platform_machine != 'x86_64' and platform_machine != 'amd64' and sys_platform != 'win32' and sys_platform != 'linux' and implementation_version != '*+cpu'"
I was attempting to debug, figuring maybe it was a hiccup with 1 marker, but I removed all markers and couldn't download the specified wheel. It refuses to update to anything but the +cpu-based packages and they are all Wins/Linux based.
Workarounds
Yeah...no workaround I know of. I see Anaconda has nightly packs avail -- donno if Poetry would be able to build 'em despite their Conda specificity? PyPi doesn't have a DB of the Nightlys AFAIK.
Only way I have gotten around it has been 1. remove Torch and point it to the correct wheel 2. use pip to circumvent the issue...this one is semi-successful. If anything, I find sometimes Poetry reverts to Torch 2.3.0, for example, then I can make things less painful by removing Torch via Pip and reinstalling the 2.4.0.devXXXXXX with Pip and that'll work until packages depending on Torch are needing to be added. Rinse and repeat.
Poetry Installation Method
other
Operating System
MacOSX 14.5
Poetry Version
1.8.1/2
Poetry Configuration
Python Sysconfig
Example pyproject.toml
No response
Poetry Runtime Logs
The text was updated successfully, but these errors were encountered: