Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement break-down mode for cross entropy calculation #1257

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

penkoba
Copy link

@penkoba penkoba commented Apr 19, 2022

This PR prevents the training process from falling down to the CPU mode, possibly happens in following case.

  • large number of objects per image (detected and ground truth)
  • large number of classes

When calculating cross entropy, the process requires a tensor with following size
nr_of_detected_objects * nr_of_ground_truth * nr_of_classes
which may explode in the above case and cause GPU OOM.

This PR implements break-down mode of cross entropy calculation, which requires only
nr_of_detected_objects * nr_of_classes
of tensor. This mode will be tried when the original code caused OOM, and still runs far faster than the CPU mode.

For testing, you can pull the following branch
https://github.com/penkoba/YOLOX/tree/break_down_cross_entropy_calc-test
and run test-train.sh, then you may see entering the break-down mode but still runs at acceptable speed.
Without this PR, the traning process falls into CPU mode very often and gets very slow.
(tested with RTX 2080 Ti)

@CLAassistant
Copy link

CLAassistant commented Apr 19, 2022

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants