DLearn, a deep learning based application that helps you better understand online video lectures by utilizing your facial expressions to determine if you are confused or stressed while watching a video, and provide you with topic recommendations that you might not have understood.
- CNN model : Costom trained Convolution Neural Net for facial expression/emotion recognition trained on FER-2013 dataset. Certain classes were merged to create new expressions so that there are 4 classes HAPPY, NEUTRAL, CONFUSED, STRESSED. Trained model for the CNN is available for download !
- Text analysis : This performs recommendations of topics depending upon the exact time when you were confused during the lecture.
- Select a video lecture that you want to learn from.
- The moment you click on play button, emotion detection model kicks in and continuously monitors your behavior.
- When the model detects that the user is stressed/confused for few continuous frames it pauses the video, analyse the transcripts of the video upto the exact time when the user seemed confused.
- The model looks few moments into the future and past and recommend topics that might have confuseed the user. The text analays model does this by learning distributed representations form the wikipedia text of the topic and comparing the similarity of the words in the transcripts.
- These topics are then presented to the user with a brief description abut each of them, giving user time to understand them before they jump to other advanced concepts deep into the video.
- Tensorflow-gpu(1.7) : for creating and training CNN.
- Flask : For deplopyment.
- OpenCV: face-detection.
- YouTube transcript API: To get transcripts for a given video.
- Python ,Javascript and HTML.
"Most technically difficult HACK" - HACKOH/IO-2019, NOV 2-3 (THE OHIO STATE UNIVERSITY)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent