quietscientist / wiggle-face

wiggle-face

This is the code for emotion recognition in infants.

Setup Instructions

1. Mount Your Tapedeck Folder:

   • Follow the [Mounting Instructions] given to you to mount your Tapedeck folder on your computer.

2. Install VS Code, Docker, and Dev Containers:

   • Download and install Visual Studio Code (VS Code).
   • Open VS Code, go to the Extensions view (Ctrl+Shift+X), and install the "Remote - Containers" extension and the "Docker" Extension.

3. Connect to Capsian:

   • Connect to capsian.med.upenn.edu using your Penn credentials on VS Code.

4. Install Docker and Dev Containers on Capsian:

   • Once connected to Capsian, ensure that Docker and the Dev Containers extension are installed and set up in VS Code.

5. Set Up Your .devcontainer Folder:

   • Create a .devcontainer folder in your project directory.
   • This folder should contain the following files:
     • devcontainer.json
     • Dockerfile
   • Fill up the files accprding to instructions

6. Configure devcontainer.json:

   • Add the appropriate mounts in the devcontainer.json file to ensure your Tapedeck folder and other necessary directories are accessible within the container.

7. Update Your Dockerfile:

   • Add the necessary commands to your Dockerfile under the RUN directive to activate the OpenMMLab environment.
   • Example:

     RUN . activate openmmlab && \
     # Add other necessary setup commands here

8. Rebuild and Reopen in Dev Container:

   • In VS Code, press Ctrl + Shift + P to open the Command Palette.
   • Select "Remote-Containers: Rebuild and Reopen in Container".

9. Run Your Code:

   • Once the development container is set up and running, you can execute your code within the container environment.

Project Structure

1. Dataloader_notebook

   • This notebook is used for loading data from a folder of .json files.
   • Outputs a .csv file with the following columns:
     • infant_id: ID of the subject
     • frame_id: Number of the frame for a certain subject ID
     • keypoint: A tuple of x and y coordinates of the keypoint
     • keypoint_score: The score of the aforementioned keypoint
     • face_part: (left eye, right eye, left brow, right brow, mouth, chin, left pupil, right pupil, and nose)
     • keypoint_index: The number of the keypoint for a certain frame

2. Data_Inspection_Emotion_Analysis

   • Takes in the dataframe generated by the previous script, rotates all the frames such that the keypoints of the nose lie on the axis, and produces 3 CSV files:
     • displacements.csv:
       • infant_id
       • frame_id
       • keypoint_score
       • face_part
       • keypoint_index
       • x_coordinate
       • y_coordinate (the keypoint column is split into these 2 columns, all the coordinates are normalized)
       • interframe_x_displacement
       • interframe_y_displacement
     • face_features.csv:
       • infant_id
       • frame_id
       • mean_y_coord_left_brow
       • mean_y_coord_right_brow
       • right_eye_eyelid_distance
       • left_eye_eyelid_distance
       • lip_distance
       • right_eye_closed
       • left_eye_closed
       • mouth_closed
     • keypoints_metrics.csv:
       • infant_id
       • keypoint_index
       • mean_x_displacement_keypoint (mean interframe x displacement per keypoint ID per infant)
       • mean_y_displacement_keypoint (mean interframe y displacement per keypoint ID per infant)
       • average_confidence_keypoint (the average score per keypoint per infant ID)
       • face_part

3. Separate_into_frames Takes in a certain number of videos from a path and separates them into frames at a specified fps. It saves the frames in .png format such that all frames of a video will be saved in a subfolder. All subfolders, titled "video_[index]", are saved in a larger folder entitled "frames_first_20_videos".

4. FER_Residual_Masking_Network This takes in all the previously generated frames and runs a vanilla FER algorithm trained on adult faces. It outputs a CSV file "labels_first_20_videos" with the following columns: infant_id, frame_number, emo_label (emotion label), angry, disgust, fear, happy, sad, surprise, neutral (scores for all emotions). If a face is not recognized, a nan value is introduced for the emotion label and all other scores.

Additional Analysis

Eyes Open/Closed - Distribution of Distance Between Eyelids Over All Frames (Histogram)

• Create a new dataframe face_feature_df with infant_id, frame_id.
• Using the original dataframe, for each infant ID and for each frame, calculate the distances between keypoint indices:
  • Right eye: 79 and 81, 78 and 82, 77 and 83; take the absolute mean of these 3 distances and introduce column right_eye_eyelid_distance to face_feature_df.
  • Left eye: 70 and 72, 69 and 73, 68 and 74; take the absolute mean of these 3 distances and introduce column left_eye_eyelid_distance to face_feature_df.
• Introduce 2 new columns to face_feature_df: left_eye_closed and right_eye_closed which should have booleans, true for closed, false for open (for each frame).
• An eye is considered closed if the distance between eyelids is close to 0.

Mouth Open/Closed - Distribution of Distance Between Upper and Lower Lip

• Using the original dataframe, for each infant ID and for each frame, calculate the distances between keypoint indices:
  • 100 and 102, 99 and 103, 98 and 104; take the absolute mean of these 3 distances and introduce column lip_distance to face_feature_df.
• Introduce a new column to face_feature_df: mouth_closed which should have boolean, true for closed, false for open (for each frame).
• The mouth is considered closed if lip_distance is close to 0.

Eyebrow Displacement - Distribution of Normalized Y Coordinates

• Using the original dataframe, for each infant ID and for each frame, calculate the mean y-coordinate for keypoint indices:
  • 34 to 42; introduce column mean_y_coord_left_brow to face_feature_df.
  • 43 to 51; introduce column mean_y_coord_right_brow to face_feature_df.
• Function to calculate the mean y-coordinate for specified keypoints.

wiggle-face's Issues

Week 4 Tasks

• Smooth the data & interpolate (mean filter + median filter)
• Find all frames where all keypoints are visible & confidence above 0.8
• Break into movement "blocks" where you get continuous good data for at least 1 second ( frame number depends on sampling rate)
• Compute new features - convexity of top/bottom lip, right/left top/bottom eyelid, left/right brow from new datset
• Use PCA to extract relevant features
• Use unsupervised model for FER on extracted features

Week 2 Tasks: Displacement of face features

Infant Features:

• Define a function that rotates so that the nose is parallel with y-axis
• Eyes open/closed
  • distribution of distance between eyelids over all frames (hist)
• Mouth open/closed
  • distribution of distance upper and lower lip
• eyebrow displacement
  • distribution of normalized y coords
• total x & y
  • displacement for all keypoints

Definition of Done:

• Function for pose rotation
• Function to calculate whether eyes are open or closed
• Function to calculate whether mouth is open or closed
• Function to calculate eyebrow displacement
• Function to calculate displacement of all keypoints

Week 3 Tasks: Testing Vanilla Algorithm and Quantifying Data

• Take 20 youtube infants
• Separate into frames, downsample to 5 fps
• Change these numbers to get ~1000 frames
• Try running vanilla adult human facial expression recognition algo: https://colab.research.google.com/drive/1ZEbJ6rJuGZ6UzmOFE1XCzSgacTUJiN-H?usp=sharing

Visualize good examples, bad examples :

• Quantify how many (eyeballed) look "correct" for each emotion

Background :

• why it’s important to understand infant facial expressions
• how SOTA in facial expression analysis is done on adults
• how we plan to combine these two things

Week 5 Tasks

• Feature scatter plots (one plot per feature, color coded by expression level)
• Documentation of process/figures etc in a google slide deck to make methods writing easier
• Bayesian surprise for expression level clusters
• Write up feature list for pre-registration
• Try to work with new data for unsupervised model (PANDA 3 Data)