An introduction to Classification Bias

Level: Beginner

Presentation: https://github.com/pyladiesams/classification-bias-beginner-apr2021/blob/master/PyLadies_Bias_2021.pdf

Workshop description

AI error and bias have critical impact on people and society. For instance, certain sub-population may be systematically misclassified, and thus discriminated. All impacted people should be able to scrutinise AI bias, even people with little AI expertise.

In this workshop, we introduce the basic concepts & methods needed to inspect classification bias, and the different types of bias to inspect. We build and compare classifiers with just a few lines of code. We use simplified visualizations that remain accessible to people with little AI expertise, yet allow to inspect all types of bias.

Notebooks

1. https://colab.research.google.com/drive/1cm5btbIiTV2663HtAeJggpRmjY4zP8di?usp=sharing
2. https://colab.research.google.com/drive/1H6mMWPJQ1f1EnkbyxVP7-6iza6c1ucij?usp=sharing
3. https://colab.research.google.com/drive/1TaEcYWRfI-uzDWUzLovXIk6Psl8rD_tC?usp=sharing

Solutions

1. https://colab.research.google.com/drive/1_56hSYtcTExXcPoXCtX05VMKtH03HCMx?usp=sharing
2. https://colab.research.google.com/drive/1Q6Nk9iROQJ9AQR-FpTTgHUG5-DSKNf1p?usp=sharing
3. https://colab.research.google.com/drive/1-I2y9J9TkLII1z2GoQD9EhYoHO5gIcKY?usp=sharing

Video record

Easy Fairish: Basics of Classification and Bias

Additional materials

Articles:

• Highly recommended! Short overview of (other) bias issues: https://www.technologyreview.com/2019/02/04/137602/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix/
• Highly recommended! Comic Book on AI Fairness: https://dataresponsibly.github.io/comics/vol2/fairness_en.pdf

Tools:

• Tool AI Fairness 360: http://aif360.mybluemix.net/data
• Tool What-If-Tool: https://pair-code.github.io/what-if-tool/
• Advanced visualizations to inspect AI models, their error/bias and their explanations: https://trustmlvis.lnu.se/

Well-known cases:

• Highly recommended! Case of bias in text processing, with python code: http://blog.conceptnet.io/posts/2017/how-to-make-a-racist-ai-without-really-trying/
• Demo of Bias in COMPAS, with python code: https://github.com/propublica/compas-analysis/
• Bias in Healthcare, through proxy variables: https://science.sciencemag.org/content/366/6464/447

Other:

• Handbook of data science: https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/Index.ipynb

Credits

This workshop was set up by @pyladiesams and @emma-ba