Segment and Complete (SAC)

Welcome to the official implementation for the CVPR2022 paper Segment and Complete: Defending Object Detectors against Adversarial Patch Attacks with Robust Patch Detection. Feel free contact me if you have any question and sorry for the waiting :)

Installation

Install the required packages by:

conda env create -f environment.yml

Usage

Download the dataset

We use 1000 images randomly selected from COCO val2017 subset for evaluation. We provide the processed data for evaluating square patch attacks of different sizes here. Please download and extract them to ./data if you want to run the evaluation pipeline. In addition, if you want to run the training, please download the train2017 images and annotations and extract them to ./data.

Run the evaluation pipeline

We provide the eval_coco.sh script for running the evaluation.

mkdir evaluation
conda activate sac
bash eval_coco.sh ${GPU_ID} ${PATCH_SIZE}

For example, to run the evaluation on gpu 0 with 100x100 patches, the last command should be:

bash eval_coco.sh 0 100

Use SAC alone

You can easily integrate SAC in your own defense codebase. For example:

from patch_detector import PatchDetector
import torch

SAC_processor = PatchDetector(3, 1, base_filter=16, square_sizes=[125, 100, 75, 50, 25], n_patch=1)
SAC_processor.unet.load_state_dict(torch.load("ckpts/coco_at.pth", map_location='cpu'))

# x: list [] of input images of shape 3xHxW with values normalized to [0, 1]
x_processed, _, _ = SAC_processor(x, bpda=True, shape_completion=True)

You can find a demo of SAC here.

Training Pipeline

1. pre-generate adversarial images

python generate_adv_data_coco.py --random

2. Train the patch segmenter on the pre-generated adv dataset

python train_coco.py -d 0 -b 16 -dir runs/coco_frcnn/pretrain -r 0.3 -n 16

3. Self adversarial training

python train_at_distributed_coco.py --multiprocessing-distributed --world-size 1 --rank 0 --dist-url 'tcp://localhost:8884' -n 16 --dir runs/coco_frcnn/self-at -e 10 -r 0.3 -b 10 --val_freq 2000 --load ${ckpt}

where ${ckpt} is path to the best checkpoint from the last step.

APRICOT-Mask dataset

We present the APRICOT-Mask dataset, which augments the APRICOT dataset with pixel-level annotations of adversarial patches. We hope APRICOT-Mask along with the APRICOT dataset can facilitate the research in building defenses against physical patch attacks, especially patch detection and removal techniques.

The detail of the APRICOT-Mask dataset can be found here. The dataset can be downloaded by the following command:

wget -r ftp://ftp.cis.jhu.edu/pub/apricot-mask

or through OneDrive.

Model checkpoints

We provide the checkpoints for segmenters for COCO and Apricot Mask. For Apricot Mask, we use 64 initial filters and downsample the image to half of the original resolution.

Citation

Please cite our paper if you find this repository helpful :)

@inproceedings{liu2022segment,
  title={Segment and Complete: Defending Object Detectors against Adversarial Patch Attacks with Robust Patch Detection},
  author={Liu, Jiang and Levine, Alexander and Lau, Chun Pong and Chellappa, Rama and Feizi, Soheil},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14973--14982},
  year={2022}
}