Variation in the benchmark

[mehdidc]

Seems there is some variation in the numbers of few datasets/models after re-running the benchmark, might be due to AMP. It happens on few datasets on zero-shot classification, diabetic retinopathy being the worse. Retrieval is fine.

See #56, where it was first detected.

[jianweif]

It could be that linear probing actually trains a classification head and then evaluates. If the training is on GPU and with pytorch, it will be non deterministic due to cudnn which is a well known issue