TAC-GAN_JeHaYaFa

This repository contains exercises and the project for the Very Deep Learning lecture at the University of Fribourg, Switzerland. Our group consists of:

• Ajayi Jesutofunmi
• Balsiger Fabian
• Hemati Hamed
• Murena Patrick
• Suter Yannick

Project: TAC-GAN

A TAC-GAN implementation in PyTorch. The original TensorFlow implementation can be found here. Following files were copied from the original implementation:

• dataprep_flowers.py: renamed to data_prep.py and modified to include the COCO dataset
• encode_text.py
• skipthoughts.py
• train.py: see the tensorflow directory

Requirements

The project requires Python 3.5.2. Install all other requirements by running pip install -r requirements.txt.

Data Preparation

We assume that we store the datasets in a Data directory in the root directory of this project throughout this data preparation.

Independent of the dataset, the skip-thought vectors need to be downloaded.

1. Download the pre-trained skip-thought vectors model into the directory Data/skipthoughts by executing:

   wget http://www.cs.toronto.edu/~rkiros/models/dictionary.txt
   wget http://www.cs.toronto.edu/~rkiros/models/utable.npy
   wget http://www.cs.toronto.edu/~rkiros/models/btable.npy
   wget http://www.cs.toronto.edu/~rkiros/models/uni_skip.npz
   wget http://www.cs.toronto.edu/~rkiros/models/uni_skip.npz.pkl
   wget http://www.cs.toronto.edu/~rkiros/models/bi_skip.npz
   wget http://www.cs.toronto.edu/~rkiros/models/bi_skip.npz.pkl
   

2. Adjust the path_to_models and path_to_labels (lines 23 and 24) in skipthoughts.py if you use another data directory than Data.

3. Install the required tokenizers with a Python script:

   import nltk
   nltk.download('punkt')
   

Flowers Dataset

To train the TAC-GAN on the flowers dataset, download the dataset by doing the following.

1. Download the flower images from here. Extract the 102flowers.tgz file and copy the extracted jpg folder to Data/datasets/flowers.

2. Download the captions from here. Extract the text_c10 folder and the allclasses.txt and paste it in the Data/datasets/flowers directory.

3. Run the flowers dataset preparation:

   python data_prep.py --data_dir=Data --dataset=flowers
   

   This script will create a set of pickled files in the Data/datasets/flowers directory which will be used during training.

ATTENTION: The TensorFlow implementation will raise an error when the flowers dataset is prepared with this implementation. It is necessary to change the names of two pickle files in the Data/datasets/flowers directory:

• flowers_tc.pkl to flower_tc.pkl
• flowers_tv.pkl to flower_tv.pkl

COCO Dataset

To train the TAC-GAN on the COCO dataset, download the dataset by doing the following.

1. Download the COCO dataset

   • 2017 Train images
   • 2017 Train/Val annotations

2. Extract both archives to the folder Data/datasets/coco.

3. Rename the folder train2017 to jpg.

4. Run make in the directory pycocotools.

5. Run the COCO dataset preparation:

   python data_prep.py --data_dir=Data --dataset=coco
   

   This script will create a set of pickled files in the Data/datasets/coco directory which will be used during training.

Experiments

COCO Dataset

We perform three experiments with the COCO dataset:

1. Full dataset with 80 categories
2. 9 animal categories
   • replace line 74 in data_prep.py with class_names = ['cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe']
3. 10 randomly selected categories
   • replace line 74 in data_prep.py with class_names = ['wine glass', 'cup', 'keyboard', 'cat', 'banana', 'surfboard', 'bus', 'truck', 'baseball glove', 'microwave']