This is the code for emotion recognition in infants.
-
Mount Your Tapedeck Folder:
- Follow the [Mounting Instructions] given to you to mount your Tapedeck folder on your computer.
-
Install VS Code, Docker, and Dev Containers:
- Download and install Visual Studio Code (VS Code).
- Open VS Code, go to the Extensions view (
Ctrl+Shift+X
), and install the "Remote - Containers" extension and the "Docker" Extension.
-
Connect to Capsian:
- Connect to
capsian.med.upenn.edu
using your Penn credentials on VS Code.
- Connect to
-
Install Docker and Dev Containers on Capsian:
- Once connected to Capsian, ensure that Docker and the Dev Containers extension are installed and set up in VS Code.
-
Set Up Your .devcontainer Folder:
- Create a
.devcontainer
folder in your project directory. - This folder should contain the following files:
devcontainer.json
Dockerfile
- Fill up the files accprding to instructions
- Create a
-
Configure devcontainer.json:
- Add the appropriate mounts in the
devcontainer.json
file to ensure your Tapedeck folder and other necessary directories are accessible within the container.
- Add the appropriate mounts in the
-
Update Your Dockerfile:
- Add the necessary commands to your
Dockerfile
under theRUN
directive to activate the OpenMMLab environment. - Example:
RUN . activate openmmlab && \ # Add other necessary setup commands here
- Add the necessary commands to your
-
Rebuild and Reopen in Dev Container:
- In VS Code, press
Ctrl + Shift + P
to open the Command Palette. - Select "Remote-Containers: Rebuild and Reopen in Container".
- In VS Code, press
-
Run Your Code:
- Once the development container is set up and running, you can execute your code within the container environment.
-
Dataloader_notebook
- This notebook is used for loading data from a folder of .json files.
- Outputs a .csv file with the following columns:
infant_id
: ID of the subjectframe_id
: Number of the frame for a certain subject IDkeypoint
: A tuple of x and y coordinates of the keypointkeypoint_score
: The score of the aforementioned keypointface_part
: (left eye, right eye, left brow, right brow, mouth, chin, left pupil, right pupil, and nose)keypoint_index
: The number of the keypoint for a certain frame
-
Data_Inspection_Emotion_Analysis
- Takes in the dataframe generated by the previous script, rotates all the frames such that the keypoints of the nose lie on the axis, and produces 3 CSV files:
- displacements.csv:
infant_id
frame_id
keypoint_score
face_part
keypoint_index
x_coordinate
y_coordinate
(thekeypoint
column is split into these 2 columns, all the coordinates are normalized)interframe_x_displacement
interframe_y_displacement
- face_features.csv:
infant_id
frame_id
mean_y_coord_left_brow
mean_y_coord_right_brow
right_eye_eyelid_distance
left_eye_eyelid_distance
lip_distance
right_eye_closed
left_eye_closed
mouth_closed
- keypoints_metrics.csv:
infant_id
keypoint_index
mean_x_displacement_keypoint
(mean interframe x displacement per keypoint ID per infant)mean_y_displacement_keypoint
(mean interframe y displacement per keypoint ID per infant)average_confidence_keypoint
(the average score per keypoint per infant ID)face_part
- displacements.csv:
- Takes in the dataframe generated by the previous script, rotates all the frames such that the keypoints of the nose lie on the axis, and produces 3 CSV files:
-
Separate_into_frames Takes in a certain number of videos from a path and separates them into frames at a specified fps. It saves the frames in .png format such that all frames of a video will be saved in a subfolder. All subfolders, titled "video_[index]", are saved in a larger folder entitled "frames_first_20_videos".
-
FER_Residual_Masking_Network This takes in all the previously generated frames and runs a vanilla FER algorithm trained on adult faces. It outputs a CSV file "labels_first_20_videos" with the following columns: infant_id, frame_number, emo_label (emotion label), angry, disgust, fear, happy, sad, surprise, neutral (scores for all emotions). If a face is not recognized, a nan value is introduced for the emotion label and all other scores.
- Create a new dataframe
face_feature_df
withinfant_id
,frame_id
. - Using the original dataframe, for each infant ID and for each frame, calculate the distances between keypoint indices:
- Right eye: 79 and 81, 78 and 82, 77 and 83; take the absolute mean of these 3 distances and introduce column
right_eye_eyelid_distance
toface_feature_df
. - Left eye: 70 and 72, 69 and 73, 68 and 74; take the absolute mean of these 3 distances and introduce column
left_eye_eyelid_distance
toface_feature_df
.
- Right eye: 79 and 81, 78 and 82, 77 and 83; take the absolute mean of these 3 distances and introduce column
- Introduce 2 new columns to
face_feature_df
:left_eye_closed
andright_eye_closed
which should have booleans, true for closed, false for open (for each frame). - An eye is considered closed if the distance between eyelids is close to 0.
- Using the original dataframe, for each infant ID and for each frame, calculate the distances between keypoint indices:
- 100 and 102, 99 and 103, 98 and 104; take the absolute mean of these 3 distances and introduce column
lip_distance
toface_feature_df
.
- 100 and 102, 99 and 103, 98 and 104; take the absolute mean of these 3 distances and introduce column
- Introduce a new column to
face_feature_df
:mouth_closed
which should have boolean, true for closed, false for open (for each frame). - The mouth is considered closed if
lip_distance
is close to 0.
- Using the original dataframe, for each infant ID and for each frame, calculate the mean y-coordinate for keypoint indices:
- 34 to 42; introduce column
mean_y_coord_left_brow
toface_feature_df
. - 43 to 51; introduce column
mean_y_coord_right_brow
toface_feature_df
.
- 34 to 42; introduce column
- Function to calculate the mean y-coordinate for specified keypoints.