Multi GPU training

[mauku]

Hello,

I am currently training to run your training script for single view pose estimation model on multi GPU. I can run the script on 1 GPU and it trains as expected. However, when i try to run the script with the steps described in section "Note on GPU parallelization", i get the following error:

File "../cosypose/cosypose/utils/multiepoch_dataloader.py", line 44, in next
idx, batch = self.dataloader_iter._get_data()
AttributeError: '_DataLoaderIter' object has no attribute '_get_data'

Would be thankfull for any help.
Thanks in advance !

[hannes56a]

Hi,
same problem. It worked for me until I updated my GPU to RTX3080ti and have to update nvidia driver, cuda, cudnn, pytorch, ...

Does anyone succesfully updated all these stuff??

[hannes56a]

Hey, I found a solution for me. I setted the "N_WORKERS" to 0 for some debug case and forgot to set it back. So the iterator runs over "_SingleProcessDataLoaderIter" instead of "_MultiProcessDataLoaderIter" (torch.utils.data.dataloader), which has no "_get_data".
So, after setting "N_WORKERS" back to standard ("min(N_CPUS - 2, 8)") this failure was gone.