CIFAR-100 with pytorch-lightning and Tensorboard logging

[Antoine101]

Hi everyone,

After a few months of using pytorch-lightning and exchanging with the team on Slack, I wanted to give back and capitalize on what I've learned, to help others.

So please find below a link to my public CIFAR-100 repo. It extends on official tutorials.
All the logging is done in Tensorboard (image samples, losses, accuracies, histograms, model, ...).

Please leave any comments if you think certain things could be improved.

CIFAR-100

Have fun!

Antoine

[rohitgr7]

1. https://github.com/Antoine101/CIFAR-100/blob/d1f9c3b2b15d5f78185153a0c09852e0c12fffc1/lightning_datamodule.py#L11-L13
   i'd recommend using self.save_hyperparameters here too to avoid explicit assignment.

2. https://github.com/Antoine101/CIFAR-100/blob/d1f9c3b2b15d5f78185153a0c09852e0c12fffc1/lightning_datamodule.py#L21-L24
   make use of stage argument to initialize the relevant datasets based on fit/test.

3. https://github.com/Antoine101/CIFAR-100/blob/d1f9c3b2b15d5f78185153a0c09852e0c12fffc1/main.py#L98-L105
   I'd recommend specifying strategy as well.

4. https://github.com/Antoine101/CIFAR-100/blob/d1f9c3b2b15d5f78185153a0c09852e0c12fffc1/lightning_module.py#L22
   since you are using ConfMat for just forward pass, I'd recommend using functional metrics here since they don't save any states and can help you save some ram/gpu memory.

5. https://github.com/Antoine101/CIFAR-100/blob/d1f9c3b2b15d5f78185153a0c09852e0c12fffc1/lightning_module.py#L25-L28
   these are not required since you have already called self.save_hyperparameters.

6. https://github.com/Antoine101/CIFAR-100/blob/d1f9c3b2b15d5f78185153a0c09852e0c12fffc1/lightning_module.py#L115-L123
   you can do this inside validaiton_step using a separate instance of ConfusionMatrix. This will ensure that only required states are saved in the memory and not the predictions/targets which can cause memory issues for huge datasets. Same for testing.

The rest looks really good!

[Antoine101]

Thank you so much Rohit for taking the time to review my code and give feedbacks!
I'll review each point and correct accordingly asap.

Cheers

Antoine

[Antoine101]

Hi @rohitgr7 ,

Following your feedbacks, I modified my code (Github up to date). Please let me know if I'm correct or if I misunderstood some stuff. Here is what I modified/implemented:

• I implemented self.save_hyperparameters() in both my LightningModule and LightningDataModule classes. And I call my params (like batch_size, learning_rate or num_workers) with self.hparams. (feedback 1 & 5)
• I implemented the stage condition check in the setup() method of the LightningDataModule class. (feedback 2) I wonder why this is better to do it that way though. Also, where should the normalization stats computation line go? I guess it should be computed once but applied both at fit and test time.
• I deleted the validation_epoch_end method with the ConfusionMatrix computation as I actually didn't need it and wasn't even logging it. (feedback 6) I actually inherited it from another classification with fewer classes. Logging a confusion matrix with 100 classes isn't useful visually.
• BUT, I modified the way I compute my confusion matrix for the test part. (feedback 4) I didn't know the distinction between class-based and functional metrics so thank you for letting me know. Now I update my confusion matrix object at each test step and compute it at the end of the test epoch. I need it here because I calculate accuracy per class for my project output, not for visual purpose.
• Finally, concerning the strategy, I don't really understand from the official doc all the options. I either run my code on the CPU or on a single GPU. So I guess I don't need this flag, no?

Additional modification not part of your feedbacks:

• I set pin_memory=True for my data loaders. However, will that cause an issue if I choose to run solely on the CPU? Should there be a condition test to set it to false in case I specify "cpu" as my accelerator?

Finally, a few more points I'd like to clear out:

• I am being prompted with the following bug: [W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.
  after an epoch has run and using CPU only. Could this be linked with the pin_memory being set to True?
• I am being warned that: Your val_dataloader's sampler has shuffling enabled, it is strongly recommended that you turn shuffling off for val/test/predict dataloaders., even though it is not the case.

Thanks again

Cheers

Antoine

[rohitgr7]

Also, where should the normalization stats computation line go? I guess it should be computed once but applied both at fit and test time.

it should go under fit.

self.validation_confmat = ConfusionMatrix(num_classes=100)

you should rename it to self.test_confmat since you are using it for testing. Might confuse someone if someone is reading your code.

cm = self.validation_confmat.compute()

after this make sure to call self.validation_confmat.reset() to reinitialize the internal states else with multiple trainer.test calls it will be an issue.

So I guess I don't need this flag, no?

yeah, it's fine.

am being prompted with the following bug: [W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.
after an epoch has run and using CPU only. Could this be linked with the pin_memory being set to True?

not sure about this.

I am being warned that: Your val_dataloader's sampler has shuffling enabled, it is strongly recommended that you turn shuffling off for val/test/predict dataloaders., even though it is not the case.

won't be an issue in the future releases since a fix was merged recently.

rest looks good 🚀

[Antoine101]

Sweet! Thanks for taking the time to review my modifications.