Multi-gpu parallel training

[yangrui12321]

I have two 1080ti, but no 3090, and the following error is reported at runtime:
Got cutlass error: Error Internal at: 363
Could not free memory: F:\Tutorial\ngp\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:458 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing
How can I configure two GPUs for parallel training? I can only run it when the aabb parameter is set to 4. Can I get the same training effect as the 3090 with two 1080ti?

[slash-under]

I don't think multi-GPU usage exists currently, but one could presumably spin up two instances on two different GPUs.

[Tom94]

This is the correct answer. To add a bit more context: we'd definitely love to have multi-GPU support for single graphics primitives in the future, but it's not a high priority at the moment.

[slash-under]

Does the underlying framework have multi-GPU support?

[Tom94]

Unfortunately no, same answer there: nice to have but not currently a high priority. (see also NVlabs/tiny-cuda-nn#63)

[slash-under]

NVIDIA/cutlass#43 is the closest upstream CUTLASS issue I can find about their routines' support. FullyFusedMLP is also per-GPU, to the best of my knowledge.

Regarding the training effect, I believe FullyFusedMLP (the faster approach) requires Tensor cores, but CUTLASS shouldn't need them.

[avinashc27]

Does Instant-ngp supports Multi-GPU Training phase [Colmap + Neural network training]? I'm bit confused as in FAQs it says only VR Rendering is supported for Multi-GPU..