Mismatched BROADCAST CPU/GPU

[x666633]

when i run horovod in tf_cnn_benchmarks, i got the error

(0) Failed precondition: Mismatched BROADCAST CPU/GPU device selection: One rank specified device CPU, but another rank specified device GPU.
[[node HorovodBroadcast_v0_cg_affine2_weights_0 (defined at :263) ]]
[[HorovodBroadcast_v0_cg_affine0_biases_0/_37]]
(1) Failed precondition: Mismatched BROADCAST CPU/GPU device selection: One rank specified device CPU, but another rank specified device GPU.
[[node HorovodBroadcast_v0_cg_affine2_weights_0 (defined at :263) ]]

[reedwm]

Unfortunately, tf_cnn_benchmarks is unmaintained and I am not very familiar with Horovod. So it is unlikely you'll get answers to Horovod issues :(