XLA_GPU and no GPU utilization

[bkj]

When I run

CUDA_VISIBLE_DEVICES=0 python fixmatch.py --filters=32 --dataset=cifar10.0@40-1 --train_dir ./experiments/fixmatch

I see the process use GPU memory in nvidia-smi, but there is 0% GPU utilization and training is super slow. When I look at the devices returned by libml/utils.py:get_available_gpus, the local_device_protos are all XLA_GPU instead of GPU. Any ideas on what might be going on here and how to fix? Presumably this is some kind of version issue?

(Apologies that this is a more general TF question, but I wasn't able to find a working fix by Googling)

[david-berthelot]

Sorry I have no idea, looks like a TensorFlow question.

[khalilsarwari]

I had a similar issue and was able to resolve it by using conda to install tensorflow (and its dependencies/cudnn etc) instead of pip