-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Model not learning #36
Comments
Hi naguileraleal, I'll need a little more information about your training and validation sets. Can you tell me the labeling distribution (positive present vs. all negative) for each set? |
Hello! About the proportion of positive vs negative pixels, the value of the |
Thanks for the information. I would recommend taking some measures to make your classes more balanced. One strategy for doing this would be to only select and annotate patches which have fibrosis regions present. Does that make sense? |
Removing all the all-background patches solved the issue. There were a lot of them (~60%). Thanks for your help! |
Hello!
First of all, thank you for this awesome project!
I'm trying to use QA to annotate fibrosis on my images and I'm not having good results. This is what my images look like
Looking at the project's Tensorboard, I'm seeing the validation loss has some epochs where it diverges and then, after some epochs, drops back to normal values. I also see a lot of epochs with very large loss values. I tried training the model with several patch sizes (256, 512, 1024) and the loss behaves the same.
These losses were obtained after training with 306 training ROIs and 181 validation ROIs, with a patch size of 256x256. Each ROI is 512x512 pixels in size. The negative class predominates over the positive class.
Train loss
Validation loss
I'm trying to find out why the model is not learning. For that reason I made some modifications to the train_model.py script, like avoiding noise/blur/scaling data augmentations, test time data augmentations, and logging each layer's gradients to tensorboard.
After avoiding noise/blur/scaling data augmentations the loss values where smaller, but the behaviour was the same as before.
This is a histogram of the gradients of the model's last layer.
I don't have a QA project that produces good segementations, so I don't know how should these gradients look like. If someone has this information it would be valuable to me.
The loss function is the
CrossEntropyLoss
QuickAnnotator/train_model.py
Line 259 in 57a1358
the last layer is a
Conv2d
QuickAnnotator/unet.py
Line 53 in 57a1358
QuickAnnotator/unet.py
Line 66 in 57a1358
so the output's values are unbounded. If the output has 0 (or very near to 0) values, as it could, the loss values should be very large, as the ones I'm seeing. Why not including a
SoftMax
activation to the last layer's output? That's my next debugging move.Pixels that belong to the "Unknown" class are assigned the -1 tag in the ground truth
QuickAnnotator/train_model.py
Line 90 in 57a1358
CrossEntropyLoss
function. What is the result of applying this function to negative values? Does it ignore these pixels? I would like these pixels not to be considered when training. Is this the effect achieved when passing a negative value to this loss?Any help is much appreciated.
The text was updated successfully, but these errors were encountered: