Skip to content
This repository has been archived by the owner on Mar 12, 2024. It is now read-only.

Machine Learning scripts for the identification of human values behind arguments.

License

Notifications You must be signed in to change notification settings

webis-de/acl22-identifying-the-human-values-behind-arguments

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

37 Commits
 
 
 
 
 
 
 
 

Repository files navigation

acl22-identifying-the-human-values-behind-arguments

license

Code and docker containers employed in the ACL'22 publication "Identifying the Human Values behind Arguments". Use this repository to reproduce the experiments of the paper or apply the classifiers to new data.

Setup

Requirements:

Download the dataset from Zenodo and extract it in the main directory:

wget https://zenodo.org/record/6855004/files/webis-argvalues-22.zip
unzip webis-argvalues-22.zip

Download the models:

wget https://zenodo.org/record/6855004/files/models.zip
unzip models.zip

Or train them yourself.

Predict

Prediction on all arguments from webis-argvalues-22/arguments.tsv with test in the Usage column, or all arguments if no such column exists.

TAG=0.1.1-nocuda # or 'TAG=0.1.1-cuda11.3' if a GPU is available
GPUS="" # or 'GPUS="--gpus=all"' to use all GPUs

# Select classifiers with --classifier: "b" for BERT, "o" for one-baseline, and "s" for SVM
docker run --rm -it --init $GPUS \
  --volume "$PWD/webis-argvalues-22:/data" \
  --volume "$PWD/models:/models" \
  --volume "$PWD:/output" \
  ghcr.io/webis-de/acl22-value-classification:$TAG \
  python predict.py --classifier bos --levels "1,2,3,4a,4b"

Evaluate

Calculate for each model the label-wise and mean Precision, Recall, F1-Score, and Accuracy.

Rscript src/R/Evaluation.R --data-dir webis-argvalues-22/ --predictions predictions.tsv

Note that the result does vary for BERT after re-training due to randomness in the training process. We had to re-train our models after the publication, so expect to get slightly different results to the publication even with the models we published. In our retries, however, the conclusions we draw in the publication were still valid.

Train Classification Models

Training on all arguments from webis-argvalues-22/arguments.tsv with train in the Usage column, or all arguments if no such column exists.

mkdir models
TAG=0.1.1-nocuda # or 'TAG=0.1.1-cuda11.3' if a GPU is available
GPUS="" # or 'GPUS="--gpus=all"' to use all GPUs

# Select classifiers with --classifier: "b" for BERT and "s" for SVM
docker run --rm -it --init $GPUS \
  --volume "$PWD/webis-argvalues-22:/data" \
  --volume "$PWD/models:/models" \
  ghcr.io/webis-de/acl22-value-classification:$TAG \
  python training.py --classifier bs --levels "1,2,3,4a,4b"

Build Docker Images

The Docker images are hosted at ghcr.io and will be pulled automatically by docker run.

If you need to change them, you can also build them:

cd src/python/
docker build -t ghcr.io/webis-de/acl22-value-classification:0.1.1-cuda11.3 --build-arg CUDA=cuda11.3 .
docker build -t ghcr.io/webis-de/acl22-value-classification:0.1.1-nocuda --build-arg CUDA=nocuda .