Thrust PTX version check

[NamjaeChoi]

Hello.

We want to use a code compiled with a lower CC (7.0) on a higher CC (8.9) GPU. This is because we have multiple HPC systems with different GPUs, and we want to build a container that is compatible for all our HPC systems. But Thrust complains that the code was not compiled with a proper CC if the major version does not match. What is the reason for this limitation and can this be lifted?

Thank you.

[jrhemstad]

Can you provide an example of what you're seeing? Is this a compile time error? Runtime? A reproducer would go a long way!

[alliepiper]

This will depend on how the executable is compiled, in particular the options listed here:

https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#options-for-steering-gpu-code-generation

It is possible to compile a suitable binary using those flags.

[alliepiper]

And if you're using CMake, these flags can be set using the CMAKE_CUDA_ARCHITECTURES option:

https://cmake.org/cmake/help/latest/variable/CMAKE_CUDA_ARCHITECTURES.html

[NamjaeChoi]

We can set the architecture specific to the GPU we are running, but that is not what we prefer. We would like to set the architecture to the lowest common version of our GPUs so that it can be run on all our GPUs. Setting multiple architectures is also limited due to one of our dependencies (Kokkos).

[jrhemstad]

The error you're seeing isn't from any requirements Thrust is enforcing. This is just propagating an error from a CUDA runtime API call.

Can you show exactly how you're specifying the architecture flags for your build? My guess is it's only embedding SASS and not PTX. Forward compatibility only works when PTX is embedded in the binary.

[NamjaeChoi]

It's an external package (PETSc) that supplies the flags, but it has a format -gencode arch=compute_xx,code=sm_xx.

[pauleonix]

@NamjaeChoi In case you are not aware of that: You can specify multiple architectures s.t. multiple versions of the compiled kernels are included in the library/executable. See Fatbinaries.

[NamjaeChoi]

@jrhemstad Thank you, your diagnosis was correct. Based on a StackOverflow thread (https://stackoverflow.com/questions/26232500/does-code-sm-x-embed-only-binary-cubin-code-or-also-ptx-code-or-both), we changed it to -arch=sm_xx, and it now runs okay.