why my test mAP is only 0.207？

[liuhaolinwen]

(ssdayolo) liuhaolin@ubuntu18:/sdb/liuhaolin/SSDA-YOLO$python -m torch.distributed.launch --nproc_per_node 2 ssda_yolov5_train.py --weights weights/yolov5l.pt --data yamls_sda/pascalvoc0712_clipart1k_VOC.yaml --name voc2clipart_ssda_960_yolov5l --img 960 --device 0,1 --batch-size 12 --epochs 200 --lambda_weight 0.005 --consistency_loss --alpha_weight 2.0

Epoch gpu_mem box obj cls total cons labels img_size
199/199 22.3G 0.07241 0.07605 0.03346 0.2014 0.01949 30 960: 100%|█████████████████████████████████████████████████| 711/711 [14:08<00:00, 1.19s/it]
Class Images Labels P R [email protected] [email protected] [email protected]:.95: 100%|████████████████████████████████████████████| 42/42 [00:06<00:00, 6.50it/s]
all 500 1526 0.61 0.338 0.396 0.208 0.207
aeroplane 500 41 0.444 0.195 0.24 0.138 0.123
bicycle 500 16 0.839 0.654 0.69 0.392 0.344
bird 500 124 0.841 0.218 0.275 0.119 0.146
boat 500 74 0.929 0.354 0.48 0.219 0.22
bottle 500 74 0.53 0.649 0.592 0.0899 0.214
bus 500 8 0.473 0.375 0.34 0.288 0.204
car 500 84 0.675 0.297 0.385 0.266 0.234
cat 500 23 0.148 0.087 0.0557 0.024 0.0248
chair 500 163 0.613 0.583 0.573 0.324 0.324
cow 500 21 0.516 0.429 0.461 0.444 0.31
diningtable 500 50 0.546 0.2 0.283 0.111 0.118
dog 500 24 0.288 0.25 0.15 0.0882 0.0844
horse 500 34 0.696 0.118 0.332 0.223 0.184
motorbike 500 10 0.773 0.2 0.338 0.248 0.241
person 500 566 0.798 0.398 0.562 0.21 0.264
pottedplant 500 94 0.667 0.468 0.559 0.357 0.32
sheep 500 33 0.535 0.212 0.221 0.0519 0.088
sofa 500 21 0.639 0.333 0.407 0.313 0.282
train 500 26 0.694 0.176 0.419 0.0956 0.156
tvmonitor 500 40 0.549 0.575 0.553 0.161 0.255
200 epochs completed in 42.124 hours.

Optimizer stripped from runs/train/voc2clipart_ssda_960_yolov5l3/weights/last_student.pt, 94.0MB
Optimizer stripped from runs/train/voc2clipart_ssda_960_yolov5l3/weights/best_student.pt, 94.0MB
Destroying process group... Done.

[hnuzhy]

We usually refer the performance of [email protected] in DAOD task. I think the lower result of yours may be caused by:

1. Did you train the VOC-style → Clipart-style CUT image style transfer model, instead of using the images generated by CycleGAN? I was sorry for not uploading the large generated results of mine.
2. During my DA training of VOC → Clipart, the whole evaluation process on Clipart1K val-set was not that stable. This may be caused by the small-scale of Clipart1K dataset (500 images in val-set with 20 object categories). I just reported the best mAP result. However, we still not always obtained higher AP values on all categories comparing with other SOTA DAOD methods. The same is true for other methods. Such a target domain dataset makes various methods objectively have great uncertainty.
3. In view of the data distribution defects mentioned above, personally, I do not recommend that you try to design new adaptive strategies by testing on Clipart1K related transfer experiments. Alternatively, other more reasonable and commonly used DAOD benchmarks settings are CityScapes → CityScapes Foggy, Cityscapes → BDD100k, Sim10k→ CityScapes and KITTI→ CityScapes. We just tested on CityScapes → CityScapes Foggy.

[liuhaolinwen]

i just used the dataset you upload and did not train the CUT image transfer model

[liuhaolinwen]

Can you tell me how to train CUT model or upload the generated rusults for me?

[hnuzhy]

Can you tell me how to train CUT model or upload the generated rusults for me?

I may give you an example of training CUT model steps about CityScapes → CityScapesFoggy:

1. Preparing sub-datasets
   (e.g., 200 images from CityScapes, and 200 images from CityScapesFoggy)

$ mkdir /datasdc/zhouhuayi/domain_adaptation/CUT/datasets/CS2CSF
$ ln -s /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/cs_da_slim \
    /datasdc/zhouhuayi/domain_adaptation/CUT/datasets/CS2CSF/trainA
$ ln -s /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapesFoggy/csf_da_slim \
    /datasdc/zhouhuayi/domain_adaptation/CUT/datasets/CS2CSF/trainB

CUT source code path : /datasdc/zhouhuayi/domain_adaptation/CUT/
source domain sub-dataset: /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/cs_da_slim
target domain sub-dataset: /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapesFoggy/csf_da_slim

2. Training the CUT model

$ cd /datasdc/zhouhuayi/domain_adaptation/CUT/
$ python train.py --dataroot ./datasets/CS2CSF --name CS2CSF_CUT_1024 --CUT_mode CUT \
    --gpu_ids 0 --batch_size 4 --preprocess scale_shortside_and_crop \
    --load_size 1024 --crop_size 320 --n_epochs 600 --n_epochs_decay 200

--preprocess scale_shortside_and_crop : using the shortside of input image to rescale it.
--load_size 1024 : the loading size of input image. All shapes of images in CityScapes are 2048 x 1024
--crop_size 320 : the cropping size for model training. The 320 is better than 512 or 256.
You can check temp result in ./checkpoints/CS2CSF_CUT_1024/web/index.html
This will take about 7.5 hours with one 3090 GPU, batchsize = 4, and running 800 epochs.

3. Testing the CUT model
   generate CityScapesFoggy-like fake CityScapes dataset
   [very slow][2975 images in train-set]

$ mkdir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF
$ mkdir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainA
$ mkdir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainB
$ ln -s /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/yolov5_format/images/train \
	/datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainA
$ ln -s /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapesFoggy/csf_da_slim \
    /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainB
$ python test.py --dataroot /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/ \
	--results_dir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/ \
	--name CS2CSF_CUT_1024 --CUT_mode CUT --phase train --preprocess none --gpu_ids 3 --num_test 3000
# the generation is time-consuming.
$ mv /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/CS2CSF_CUT_1024/train_latest/images/fake_B/* \
	/datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/yolov5_format_csf/images/train
$ rm -rf /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/

[slow][500images in val-set]

$ mkdir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF
$ mkdir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainA
$ mkdir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainB
$ ln -s /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/yolov5_format/images/val \
	/datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainA
$ ln -s /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapesFoggy/csf_da_slim \
    /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/trainB
$ python test.py --dataroot /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/ \
	--results_dir /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/ \
	--name CS2CSF_CUT_1024 --CUT_mode CUT --phase train --preprocess none --gpu_ids 3 --num_test 510
# the generation is time-consuming.
$ mv /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/CS2CSF_CUT_1024/train_latest/images/fake_B/* \
	/datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/yolov5_format_csf/images/val
$ rm -rf /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/images_da_CSF/

move GT labels

$ cp -r /datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/yolov5_format/labels \
	/datasdc/zhouhuayi/dataset/domain_adaptation/CityScapes/yolov5_format_csf/

Please refer the official code link of CUT https://github.com/taesungp/contrastive-unpaired-translation for more details.