Getting significantly worse performance using BackboneFinetuning Callback

[sandeepmukh]

Hi! I am new to lightning, but have used pytorch a fair amount in the past. I'm trying to use the BackboneFinetuning callback, but I'm getting terrible results doing so verus just manually enabling gradients in the on_validation_epoch_end hook.

My architecture is roughly:
backbone: ResNet50
Classification Head: SelfAttention blocks

This is my BackboneFinetuning:
BackboneFinetuning(unfreeze_backbone_at_epoch=3, backbone_initial_ratio_lr=1, verbose=True),
I have tried using different backbone_initial_ratio_lr with the same poor results.

This approach is what is currently working:

if self.current_epoch == self.transfer_epoch - 1:
            for param in self.backbone.parameters():
                param.requires_grad = True
            self.backbone.train()

Am I doing something obviously wrong?

Here are some reference training curves:
With bespoke implementation is green and using the callback is brown.