Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A lot of warnings in instance segmentation training #3

Open
ghazni123 opened this issue Nov 23, 2020 · 1 comment
Open

A lot of warnings in instance segmentation training #3

ghazni123 opened this issue Nov 23, 2020 · 1 comment

Comments

@ghazni123
Copy link

If you do not know the root cause of the problem / bug, and wish someone to help you, please
post according to this template:

Instructions To Reproduce the Issue:

  1. what changes you made (git diff) or what code you wrote
<put diff or code here>
  1. what exact command you run:
    I have tried below command

python train_net.py --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml --num-gpus 2 SOLVER.IMS_PER_BATCH 12 SOLVER.BASE_LR 0.005 MODEL.WEIGHTS ./imgnet_trained_model/r2_101.pkl

r2_101.pkl: file downloaded from link in this repo

However there are lots and lots of warnings. sounds like not much was loaded. please find attached for warnings
training_warnings.log

I have also tried standard command (which downloads the weights from Model.Zoo of FAIR) but the results are same.

python train_net.py --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml --num-gpus 2 SOLVER.IMS_PER_BATCH 12 SOLVER.BASE_LR 0.005

Could you please advise? Thank you.

  1. what you observed (including full logs):
<put logs here>
  1. please also simplify the steps as much as possible so they do not require additional resources to
    run, such as a private dataset.

Expected behavior:

I think even with res2net backbone, the training should have started clean.

If there are no obvious error in "what you observed" provided above,
please tell us the expected behavior.

If you expect the model to converge / work better, note that we do not give suggestions
on how to train a new model.
Only in one of the two conditions we will help with it:
(1) You're unable to reproduce the results in detectron2 model zoo.
(2) It indicates a detectron2 bug.

Environment:

Provide your environment information using the following command:

wget -nc -q https://github.com/facebookresearch/detectron2/raw/master/detectron2/utils/collect_env.py && python collect_env.py

If your issue looks like an installation issue / environment issue,
please first try to solve it yourself with the instructions in
https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md#common-installation-issues

@ghazni123 ghazni123 changed the title Please read & provide the following A lot of warnings in instance segmentation training Nov 23, 2020
@ghazni123
Copy link
Author

Environment:
wget -nc -q https://github.com/facebookresearch/detectron2/raw/master/detectron2/utils/collect_env.py && python collect_env.py


sys.platform linux
Python 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:45:15) [GCC 7.5.0]
numpy 1.19.2
detectron2 0.1.1 @/home/mgsaeed/sg1tb/model-factory/res2net/Res2Net-detectron2/detectron2
detectron2 compiler GCC 7.5
detectron2 CUDA compiler 10.2
detectron2 arch flags sm_75
DETECTRON2_ENV_MODULE
PyTorch 1.5.0 @/home/mgsaeed/anaconda3/envs/detectron2_res2net/lib/python3.7/site-packages/torch
PyTorch debug build False
CUDA available True
GPU 0,1 GeForce RTX 2080 Ti
CUDA_HOME /usr/local/cuda-10.2
NVCC Cuda compilation tools, release 10.2, V10.2.89
Pillow 8.0.1
torchvision 0.6.0a0+82fd1c8 @/home/mgsaeed/anaconda3/envs/detectron2_res2net/lib/python3.7/site-packages/torchvision
torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75
fvcore 0.1.2.post20201122
cv2 4.4.0


PyTorch built with:

  • GCC 7.3
  • C++ Version: 201402
  • Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  • Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  • OpenMP 201511 (a.k.a. OpenMP 4.5)
  • NNPACK is enabled
  • CPU capability usage: AVX2
  • CUDA Runtime 10.2
  • NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  • CuDNN 7.6.5
  • Magma 2.5.2
  • Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant