Mitigating the class dependency bias of data augmentation

This repository hosts the code used for our proposed solution to mitigate the class dependecy bias introduced when applying data augmentation during training as was shown in this paper.

Code is based to run on Google Colaboratory and need to have inside the root folder of Google Drive the TinyImageNet 200 dataset.

You can find the final report and the paper produced here

Architecture

In particular our implementation is based on an architecture formed by two different sub-nets, one trained solely with augmented data and the other trained solely on the original data. This approach aims to give more weight on one of the two nets for each specific image based on which of the two nets has a superior performance with that specific data.

In this specific experiment due to the computational limitation and time constraint, we used as base net the ResNet18.

Results

It is possible to see the complete results and consideration inside our final relation. We briefly report here the results we had with different indexes to evaluate the performances of the various implementations.

ResNet18

+----------+----------+---------+---------+-----------+----------+
|   Crop % |   Mean % |     Var |     Std |   CVaR 5% |   Top 5% |
|----------+----------+---------+---------+-----------+----------|
| 0.08     |    46.5  | 399.508 | 19.9877 |       8.4 |     88.8 |
| 0.156667 |    48.54 | 404.049 | 20.101  |      13.6 |     86.4 |
| 0.233333 |    49.5  | 404.01  | 20.1    |      10.4 |     84.8 |
| 0.31     |    50.2  | 404.864 | 20.1212 |      12.8 |     88   |
| 0.386667 |    49.82 | 443.304 | 21.0548 |       8.8 |     87.6 |
| 0.463333 |    50.38 | 386.749 | 19.6659 |      14.8 |     84.4 |
| 0.54     |    50.78 | 397.7   | 19.9424 |      14.4 |     90   |
| 0.616667 |    50.22 | 419.489 | 20.4814 |      10.4 |     85.6 |
| 0.693333 |    48.86 | 396.523 | 19.9129 |      10   |     89.2 |
| 0.77     |    50.02 | 356.502 | 18.8813 |      13.6 |     87.6 |
| 0.846667 |    51.5  | 388.975 | 19.7224 |      13.2 |     87.6 |
| 0.923333 |    50    | 353.93  | 18.813  |      16.8 |     86.8 |
| 1        |    48.8  | 382.07  | 19.5466 |      13.6 |     86.4 |
+----------+----------+---------+---------+-----------+----------+

ResNet34

+----------+----------+---------+---------+-----------+----------+
|   Crop % |   Mean % |     Var |     Std |   CVaR 5% |   Top 5% |
|----------+----------+---------+---------+-----------+----------|
| 0.08     |    49.9  | 373.538 | 19.3271 |      12.8 |     86.8 |
| 0.156667 |    50.28 | 342.755 | 18.5137 |      14.4 |     82.8 |
| 0.233333 |    51.96 | 355.858 | 18.8642 |      14.8 |     83.6 |
| 0.31     |    53.5  | 386.322 | 19.6551 |      14.4 |     88.4 |
| 0.386667 |    50.32 | 422.008 | 20.5428 |      13.2 |     87.2 |
| 0.463333 |    51.38 | 376.136 | 19.3942 |      14   |     86.4 |
| 0.54     |    52.58 | 343.702 | 18.5392 |      14.4 |     83.6 |
| 0.616667 |    52.96 | 360.24  | 18.98   |      14.8 |     87.2 |
| 0.693333 |    52.48 | 358.683 | 18.9389 |      17.6 |     86.4 |
| 0.77     |    53.42 | 376.426 | 19.4017 |      16   |     89.6 |
| 0.846667 |    51.82 | 347.706 | 18.6469 |      14.8 |     87.6 |
| 0.923333 |    51.56 | 335.243 | 18.3096 |      16.4 |     87.2 |
| 1        |    51.38 | 319.372 | 17.871  |      15.2 |     82   |
+----------+----------+---------+---------+-----------+----------+

Combined ResNet18 from scratch

+----------+----------+---------+---------+-----------+----------+
|   Crop % |   Mean % |     Var |     Std |   CVaR 5% |   Top 5% |
|----------+----------+---------+---------+-----------+----------|
| 0.08     |    44.4  | 344.925 | 18.5721 |       9.6 |     81.2 |
| 0.156667 |    39.42 | 333.411 | 18.2595 |       8.4 |     75.2 |
| 0.233333 |    34.42 | 329.149 | 18.1425 |       2.8 |     76.4 |
| 0.31     |    37.82 | 408.41  | 20.2091 |       3.6 |     82.4 |
| 0.386667 |    42.22 | 337.64  | 18.375  |       9.2 |     79.6 |
| 0.463333 |    34.46 | 359.024 | 18.9479 |       4.4 |     75.6 |
| 0.54     |    49.26 | 409.661 | 20.2401 |      12.4 |     87.2 |
| 0.616667 |    43.14 | 445.568 | 21.1085 |       6.8 |     87.2 |
| 0.693333 |    35.56 | 431.564 | 20.7741 |       4   |     78.4 |
| 0.77     |    34.54 | 354.601 | 18.8309 |       1.6 |     73.2 |
| 0.846667 |    48.8  | 411.176 | 20.2775 |      10.8 |     86   |
| 0.923333 |    36.02 | 459.175 | 21.4284 |       2   |     76.4 |
| 1        |    38.08 | 336.396 | 18.3411 |       5.6 |     76.4 |
+----------+----------+---------+---------+-----------+----------+

Combined ResNet18 pre-trained

+----------+----------+---------+---------+-----------+----------+
|   Crop % |   Mean % |     Var |     Std |   CVaR 5% |   Top 5% |
|----------+----------+---------+---------+-----------+----------|
| 0.08     |    49.4  | 345.045 | 18.5754 |      15.2 |     85.6 |
| 0.156667 |    50.24 | 360.465 | 18.9859 |      16.8 |     86.8 |
| 0.233333 |    50.76 | 341.932 | 18.4914 |      17.2 |     86.8 |
| 0.31     |    50.76 | 358.656 | 18.9382 |      16   |     87.2 |
| 0.386667 |    51.52 | 339.547 | 18.4268 |      16.8 |     86.4 |
| 0.463333 |    51.22 | 355.489 | 18.8544 |      16.8 |     86.4 |
| 0.54     |    51.56 | 344.248 | 18.5539 |      17.2 |     85.6 |
| 0.616667 |    52.58 | 348.205 | 18.6602 |      17.6 |     88.8 |
| 0.693333 |    52.46 | 345.918 | 18.5989 |      16.4 |     88.4 |
| 0.77     |    52.8  | 334.472 | 18.2886 |      17.6 |     86.8 |
| 0.846667 |    52.98 | 348.542 | 18.6693 |      17.6 |     88   |
| 0.923333 |    53.2  | 350.231 | 18.7145 |      16.4 |     88.4 |
| 1        |    53.24 | 319.098 | 17.8633 |      19.2 |     87.2 |
+----------+----------+---------+---------+-----------+----------+