Poetry fails to resolve dependencies that pull from a source

[brian316]

Description

I have a pyproject.toml that pulls another remote package that has a source for torch-scatter

the torch scatter implementation is as such

[tool.poetry.dependencies]
python = "^3.10"
torch-scatter = { version = "=2.1.2+pt22cpu", source = "scatter-cpu" }

[[tool.poetry.source]]
name = "scatter-cpu"
url = "https://data.pyg.org/whl/torch-2.2.0%2Bcpu.html"

if i build that project locally it installs fine. but if I call to build from another pyproject it gives error Repository "scatter-cpu" does not exist. when attempting to do a poetry lock

Workarounds

right now i have to add the source from the dependency:

[[tool.poetry.source]]
name = "scatter-cpu"
url = "https://data.pyg.org/whl/torch-2.2.0%2Bcpu.html"

to my top level pyproject.toml or it wont build.

Poetry Installation Method

pipx

Operating System

Fedora 38

Poetry Version

1.8.2

Poetry Configuration

cache-dir = "/home/dev/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.scatter-cpu.url = "https://data.pyg.org/whl/torch-2.2.0%2Bcpu.html"
repositories.torch-cpu.url = "https://download.pytorch.org/whl/cpu"
repositories.torch-cu118.url = "https://download.pytorch.org/whl/cu118"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/dev/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

[tool.poetry.dependencies]
python = "^3.10"
torch-scatter = { version = "=2.1.2+pt22cpu", source = "scatter-cpu" }

[[tool.poetry.source]]
name = "scatter-cpu"
url = "https://data.pyg.org/whl/torch-2.2.0%2Bcpu.html"

Poetry Runtime Logs

Stack trace:

  4  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
      152│ 
      153│         try:
    → 154│             result = resolve_version(self._package, self._provider)
      155│ 
      156│             packages = result.packages

  3  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
      173│             while next is not None:
      174│                 self._propagate(next)
    → 175│                 next = self._choose_package_version()
      176│ 
      177│             return self._result()

  1  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
      512│             package = locked
      513│ 
    → 514│         package = self._provider.complete_package(package)
      515│ 
      516│         conflict = False

  OverrideNeeded

  ({Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx, source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (>=1.7.1)>}}, {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx', source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (<empty>)>}}, {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx', source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (>=1.7.1+cu118)>}}, {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx', source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (>=1.7.1+cpu)>}})

  at ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  4  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
      152│ 
      153│         try:
    → 154│             result = resolve_version(self._package, self._provider)
      155│ 
      156│             packages = result.packages

  3  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
      173│             while next is not None:
      174│                 self._propagate(next)
    → 175│                 next = self._choose_package_version()
      176│ 
      177│             return self._result()

  1  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
      512│             package = locked
      513│ 
    → 514│         package = self._provider.complete_package(package)
      515│ 
      516│         conflict = False

  OverrideNeeded

  {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx, source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (<empty>)>, 'torch-scatter': <URLDependency torch-scatter (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_scatter-2.1.2%2Bpt22cpu-cp310-cp310-linux_x86_64.whl>}}

  at ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  4  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
      152│ 
      153│         try:
    → 154│             result = resolve_version(self._package, self._provider)
      155│ 
      156│             packages = result.packages

  3  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
      173│             while next is not None:
      174│                 self._propagate(next)
    → 175│                 next = self._choose_package_version()
      176│ 
      177│             return self._result()

  1  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
      512│             package = locked
      513│ 
    → 514│         package = self._provider.complete_package(package)
      515│ 
      516│         conflict = False

  OverrideNeeded

  {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx, source_reference='main', source_resolved_reference=', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (<empty>)>, 'torch-scatter': <URLDependency torch-scatter (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_scatter-2.1.2%2Bpt22cpu-cp310-cp310-linux_x86_64.whl>, 'torch-cluster': <URLDependency torch-cluster (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_cluster-1.6.3-cp310-cp310-macosx_11_0_x86_64.whl>}}

  at ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  4  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
      152│ 
      153│         try:
    → 154│             result = resolve_version(self._package, self._provider)
      155│ 
      156│             packages = result.packages

  3  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
      173│             while next is not None:
      174│                 self._propagate(next)
    → 175│                 next = self._choose_package_version()
      176│ 
      177│             return self._result()

  1  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
      512│             package = locked
      513│ 
    → 514│         package = self._provider.complete_package(package)
      515│ 
      516│         conflict = False

  OverrideNeeded

  {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx', source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (<empty>)>, 'torch-scatter': <URLDependency torch-scatter (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_scatter-2.1.2%2Bpt22cpu-cp310-cp310-linux_x86_64.whl>, 'torch-cluster': <URLDependency torch-cluster (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_cluster-1.6.3-cp310-cp310-macosx_11_0_x86_64.whl>}, Package('gt4sd-inference-torch-drug', '0.1.0', source_type='directory', source_url='/home/dev/.cache/pypoetry/virtualenvs/openad-model-generation-service-W9SyLlNI-py3.10/src/Open-AD-Model-Service/openad-model-inference/gt4sd_torch'): {'torch': <Dependency torch (<empty>)>}}

  at ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  4  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
      152│ 
      153│         try:
    → 154│             result = resolve_version(self._package, self._provider)
      155│ 
      156│             packages = result.packages

  3  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
       16│     solver = VersionSolver(root, provider)
       17│ 
    →  18│     return solver.solve()
       19│ 

  2  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
      173│             while next is not None:
      174│                 self._propagate(next)
    → 175│                 next = self._choose_package_version()
      176│ 
      177│             return self._result()

  1  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
      512│             package = locked
      513│ 
    → 514│         package = self._provider.complete_package(package)
      515│ 
      516│         conflict = False

  OverrideNeeded

  {Package('gt4sd-inference-all', '0.1.0', source_type='git', source_url='xxxx', source_reference='main', source_resolved_reference='', source_subdirectory='openad-model-inference/gt4sd_inference_all'): {'torch': <Dependency torch (<empty>)>, 'torch-scatter': <URLDependency torch-scatter (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_scatter-2.1.2%2Bpt22cpu-cp310-cp310-linux_x86_64.whl>, 'torch-cluster': <URLDependency torch-cluster (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_cluster-1.6.3-cp310-cp310-macosx_11_0_x86_64.whl>}, Package('gt4sd-inference-torch-drug', '0.1.0', source_type='directory', source_url='/home/dev/.cache/pypoetry/virtualenvs/openad-model-generation-service-W9SyLlNI-py3.10/src/Open-AD-Model-Service/openad-model-inference/gt4sd_torch'): {'torch': <Dependency torch (<empty>)>, 'torch-cluster': <URLDependency torch-cluster (<empty>) @ https://data.pyg.org/whl/torch-2.2.0%2Bcpu/torch_cluster-1.6.3-cp310-cp310-macosx_11_0_x86_64.whl>}}

  at ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:


  Stack trace:

  31  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  30  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  29  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  28  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  27  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  26  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│ 
     → 117│         return self.execute(io) or 0
       118│ 
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  25  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/cleo/commands/command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  24  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/console/commands/lock.py:55 in handle
        53│         self.installer.lock(update=not self.option("no-update"))
        54│ 
     →  55│         return self.installer.run()
        56│ 

  23  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│ 
     → 104│         return self._do_install()
       105│ 
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

  22  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/installation/installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

  21  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│ 

  ...  Previous 2 frames repeated 3 times

  14  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:158 in _solve
       156│             packages = result.packages
       157│         except OverrideNeeded as e:
     → 158│             return self._solve_in_compatibility_mode(e.overrides)
       159│         except SolveFailure as e:
       160│             raise SolverProblemError(e)

  13  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:132 in _solve_in_compatibility_mode
       130│             )
       131│             self._provider.set_overrides(override)
     → 132│             _packages, _depths = self._solve()
       133│             for index, package in enumerate(_packages):
       134│                 if package not in packages:

  12  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:158 in _solve
       156│             packages = result.packages
       157│         except OverrideNeeded as e:
     → 158│             return self._solve_in_compatibility_mode(e.overrides)
       159│         except SolveFailure as e:
       160│             raise SolverProblemError(e)

  11  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:132 in _solve_in_compatibility_mode
       130│             )
       131│             self._provider.set_overrides(override)
     → 132│             _packages, _depths = self._solve()
       133│             for index, package in enumerate(_packages):
       134│                 if package not in packages:

  10  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
       152│ 
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│ 
       156│             packages = result.packages

   9  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

   8  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│ 
       177│             return self._result()

   7  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:493 in _choose_package_version
       491│             return is_specific_marker, preference, -num_packages
       492│ 
     → 493│         dependency = min(unsatisfied, key=_get_min)
       494│ 
       495│         locked = self._provider.get_locked(dependency)

   6  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:480 in _get_min
       478│ 
       479│             num_packages = len(
     → 480│                 self._dependency_cache.search_for(
       481│                     dependency, self._solution.decision_level
       482│                 )

   5  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:112 in search_for
       110│         # We could always use dependency.without_features() here,
       111│         # but for performance reasons we only do it if necessary.
     → 112│         packages = self._search_for_cached(
       113│             dependency.without_features() if dependency.features else dependency, key
       114│         )

   4  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/mixology/version_solver.py:93 in _search_for
        91│         # additional results this time.
        92│         if not packages:
     →  93│             packages = self._provider.search_for(dependency)
        94│ 
        95│         return packages

   3  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/puzzle/provider.py:300 in search_for
       298│             return PackageCollection(dependency, packages)
       299│ 
     → 300│         packages = self._pool.find_packages(dependency)
       301│ 
       302│         packages.sort(

   2  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/repositories/repository_pool.py:212 in find_packages
       210│         repository_name = dependency.source_name
       211│         if repository_name:
     → 212│             return self.repository(repository_name).find_packages(dependency)
       213│ 
       214│         packages: list[Package] = []

   1  ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/repositories/repository_pool.py:136 in repository
       134│ 
       135│     def repository(self, name: str) -> Repository:
     → 136│         return self._get_prioritized_repository(name).repository
       137│ 
       138│     def get_priority(self, name: str) -> Priority:

  IndexError

  Repository "scatter-cpu" does not exist.

  at ~/.local/share/pipx/venvs/poetry/lib64/python3.11/site-packages/poetry/repositories/repository_pool.py:145 in _get_prioritized_repository
      141│     def _get_prioritized_repository(self, name: str) -> PrioritizedRepository:
      142│         name = name.lower()
      143│         if self.has_repository(name):
      144│             return self._repositories[name]
    → 145│         raise IndexError(f'Repository "{name}" does not exist.')
      146│ 
      147│     def add_repository(
      148│         self,
      149│         repository: Repository,

[dimbleby]

this is expected, source information cannot be transferred via the standards-defined metadata.

please close