This repository contains the codebase for the thesis "An Explainable Deep Learning Baseline for Iconography Research in Artworks" written by Christopher Buch Madsen while attending the Bachelor of AI at the University of Amsterdam. The thesis was delivered 29 January, 2021.
Python >= 3.9
PyTorch >= 1.7.1+cu110
Numpy
Pandas
Matplotlib
Sklearn (>= 0.24.1)
ipynb (for examples)
- Clone this repository.
- Download the ArtDL data set at:
http://www.artdl.org/
Unzip the files and place the "DEVKitArt" folder in the folder prior to this repository. - Create a folder named "DataFolders" at the same destination as DEVKitArt
The directory tree should at this point look like:
|
|───DEVKitArt
|
|───DataFolders
|
|───main <--- the project repository
|
...
- Run the following python files from main/preprocessing consecutively:
- seclude_img.py
- make_class_folders.py
- sort_data_by_folders.py
- sort_data_by_folder_no_dup.py
This will set up the data in sorted folders, necessary for the project.
The updated directory tree:
|
|───DEVKitArt
|
|───DataFolders
| |───data_by_class
| |───data_by_class_no_dup
| |───test_folder
| | |
| | |───test
| |
| |───train_folder
| | |
| | |───train
| |
| |───val_folder
| | |
| | |───val
|
|───main <--- the project repository
|
...
To extract the class activation mappings (CAMs) for the test set with the VGG-16 models, run the following python script and answer the prompt:
python vgg_cam.py
To extract the CAMs from the ArtDL model, the following python script can be run:
python artdl_cam.py
To classify the images in the test set with the VGG-16 models, run the following script and answer the prompt:
python vgg_classify.py
To classify with the ArtDL model use:
python artdl_classify.py
Running these scripts will save the predictions in the best_vgg_(model_number)_strategy
folders for the VGG-16 model and evaluation_files
folder for the ArtDL model.
To show the evaluation metrics for a model, run the following script and answer the prompt:
python eval.py
This will generate the confusion matrices used for precision, recall and f1-score from the thesis.
A simple example notebook for making a classification of a single image and extracting the CAM has been provided in the jupyter notebook vgg_cam_example.ipynb, for a better overview. (Install jupyter notebook with pip install notebook, open it through the command line with "jupyter notebook" and navigate to the .ipynb provided)
If one desires to train the VGG-16 model using one of the strategies from the thesis, it is possible through the use of the following python scripts:
python train_vgg.py
This will run the training for the amount of epochs stated in the file. Training the models with 200 epochs will take approximately 14 hours for strategy 2, and approximately 84 hours for strategy 1 and 3 (on computer architecture similar to the one used in the thesis).
After the model(s) is done training, it will save the weights and training metrics in the best_vgg_(model number)_strategy
folder.
architecture
is the location of the model architecture files.
best_vgg_(model number)__strategy
contain the final models (weights) from the experiments of the thesis. It also contains the training metrics as well as classifications made with vgg_classify.py
evaluation_files
is the destination for the CAMs extracted with vgg_cam.py and artdl_cam.py, it also contains the classifications made by the ArtDL model, and .csv files for the final predictions used for the CAMs.
artdl_model
contains the weights for the ResNet50 model provided by the ArtDL project.
preprocessing
contains files used for data preprocessing and creating of data folders.
sets
contains text files for the training/validation/test sets and the lists of classes.
test-data
contains images to be used in the example notebook
torch_mods
contains files which modfiy Torch classes.
For artdl_cam.py, artdl_classify.py, train_vgg.py, vgg_cam.py, vgg_classify.py, eval.py
see the descriptions above.
data_prep_example.ipynb
is a notebook used for producing the preprocessing example shown in the thesis
training_graphs.ipynb
is a notebook used for producing the training metrics graphs shown in the thesis.
All rights of the ArtDL project and model go to Milani et al., their repository can be found at: https://github.com/iFede94/ArtDL
See the ArtDL_README.md