Baby Jokes Video Analysis

Caspar Addyman [email protected]

A demonstration project using machine learning models to analyse dataset of videos of parents demonstrating jokes to babies. This dataset was assembled for Sage Ethical AI hackathon 2023. It serves as a small test case to explore challenges with machine learning models of parent child interactions. You can watch a video motivating the project here Sage Hackathon 2023 - PCI Video Analysis 6m20

Dataset

A small test dataset is provided in the LookitLaughter.test folder. It consists of 54 videos of parents demonstarting simple jokes to their babies. Metadata is provided in _LookitLaughter.xlsx. Each video shows one joke from a set of five possibilities [Peekaboo,TearingPaper,NomNomNom,ThatsNotAHat,ThatsNotACat]. For each joke parents rated how funny the child found it [Not Funny, Slightly Funny, Funny, Extremely Funny] and whether they laughed [Yes, No] A larger dataset with 1425 videos is available on request.

Code

All notebooks and supporting code are in the code folder. The numbered notebooks should be run in order to process the data, train the models and generate the results.n

#TODO - visualise data #TODO - build models & analysis

Installation / Key Requirements

This project makes use of the following libraries and versions:

• Python 3.11
• Pytorch 2.1.0 (for YOLOv8, deepface, whisper)
• ultralytics 8.0 (wrapper for YOLOv8 object detection model)
• deepface 0.0.68 (Facial Expression Recognition)
• speechbrain 0.5 (Speech Recognition)
• openai-whisper (OpenAI's Whisper speech recognition -open source version)

Installing with Conda

A Conda environment.yml file is provided but dependencies are complex so can fail to install in a single step. The culprit seems to be the pytorch dependencies. So instead run the follow commands in the terminal.

1. Create a new Python 3.11 environment

conda create --name "babyjokes" python=3.11

2. Activate the environment

conda activate babyjokes

3. Install PyTorch Advisable to follow the instructions at pytorch.org to get the correct version for your system.
4. Add the other dependencies.
   Run the following command from the root directory of this project.

conda env update --file environment.yml

Installing with Pip

We also provide a pip requirements.txt file. This should work but has not been tested. We recommend following similar steps to the conda installation above.

1. Create a new python 3.11 environment.
2. Install PyTorch
3. Installing the other dependencies:

pip install -r requirements.txt

If you get this working, please let us know what you did (and what OS you are using) so we can update this README.

Sage Hackathon

Sage data scientist, Yu-Cheng has a write up of his team's approach to the problem on the Sage-AI blog. Quantifying Parent-Child Interactions: Advancing Video Understanding with Multi-Modal LLMs Repositories from the hackathon are found here:

• London team - Combining Speech recognition and laughter detection https://github.com/chilledgeek/ethical_ai_hackathon_2023
• US team - Interpreting Parent laughter with VideoLLama https://github.com/yutsai84/Ask-Anything