Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add early stopping with warmup. Remove mandatory background label in semantic segmentation task #3515

Merged
merged 13 commits into from
May 28, 2024

Conversation

kprokofi
Copy link
Contributor

@kprokofi kprokofi commented May 17, 2024

Summary

Motivation:

  1. Early stopping from lightning doesn't have a warmup parameter and it leads to underfitting when training on small Geti target datasets. I added two thresholds to have invariant for small datasets that can fit in one batch and for larger data

  2. Inserting a background label actually can lead to class mismatches for the Segmentation head, sometimes datasets don't have background classes at all. I tried to divide the Geti case with polygons and ordinary training from file masks. When dm_dataset represents annotations with Polygons -> we insert a background label and create a head with num_classes + 1. For other cases, dataset_meta.json should explicitly reflect all classes for training. So, the number of indices in masks == num categories in dataset_meta.json.
    Before these changes, if we add a background class to dataset_meta.json, the training will fail, because OTX deletes the background class during prefiltering. Now, the background class must be presented in a metadata file when needed.

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have ran e2e tests and there is no issues.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

src/otx/core/data/dataset/segmentation.py Outdated Show resolved Hide resolved
src/otx/algo/callbacks/adaptive_early_stopping.py Outdated Show resolved Hide resolved
@kprokofi kprokofi requested review from vinnamkim and eunwoosh and removed request for vinnamkim May 23, 2024 15:33
@github-actions github-actions bot added the TEST Any changes in tests label May 23, 2024
Copy link

codecov bot commented May 23, 2024

Codecov Report

Attention: Patch coverage is 65.38462% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 82.86%. Comparing base (ce2d99f) to head (3beac1b).
Report is 3 commits behind head on develop.

Files Patch % Lines
src/otx/core/data/dataset/segmentation.py 55.55% 4 Missing ⚠️
src/otx/algo/callbacks/adaptive_early_stopping.py 81.81% 2 Missing ⚠️
src/otx/core/data/pre_filtering.py 33.33% 2 Missing ⚠️
src/otx/core/types/label.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3515      +/-   ##
===========================================
- Coverage    82.99%   82.86%   -0.14%     
===========================================
  Files          253      254       +1     
  Lines        25163    24945     -218     
===========================================
- Hits         20885    20670     -215     
+ Misses        4278     4275       -3     
Flag Coverage Δ
py310 82.24% <65.38%> (-0.27%) ⬇️
py311 82.74% <65.38%> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/otx/core/data/dataset/segmentation.py Outdated Show resolved Hide resolved
tests/unit/core/types/test_label.py Outdated Show resolved Hide resolved
eunwoosh
eunwoosh previously approved these changes May 27, 2024
@kprokofi kprokofi enabled auto-merge (squash) May 27, 2024 14:25
Copy link
Contributor

@harimkang harimkang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kprokofi kprokofi merged commit 80acb86 into develop May 28, 2024
14 of 15 checks passed
@kprokofi kprokofi deleted the kp/fix_segm_dataset branch May 28, 2024 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OTX 2.0 For OTX v2.0 TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants