emrnitk / gesturecontrol

Using gestures to control media players and more. Image processing workshop project for EmR club, NIT Kurukshetra

Python module/script allowing a user to control any media player using hand gestures and face tracking.

This project is being made and maintained by the Embedded Systems and Robotics Club, NIT Kurukshetra (EmR) for its Image processing workshop. However, members outside the club are welcome to contribute (refer to contribution guidelines)

Features

• Hand gesture recognition and processing to allow user to play/pause, seek, change volume.
• Face tracking for automatic play/pause based on whether user is viewing media/is present.
• Functions to allow tuning of the image processing algorithms.
• Use of auto keystroke and clicker modules to allow application to any media player.

Installation and Usage

Dependecies

• opencv-python
• mediapipe
• PyAutoGUI
• Numpy

Command to run the python script

 python Gesture_control.py

Project Explanation

The project working can be summarized into two parts:

• Hand Motion Tracking:

  • Mask of hand is created to detect the hand.

       lower = np.array([hue_min, sat_min, val_min])
       upper = np.array([hue_max, sat_max, val_max])
       mask = cv2.inRange(imgHSV, lower, upper)

  • Contours of mask is found out using findContours(). The contour with maximum area is selected as the contour of hand as other contours are due to noise.

       contours, heirarchy = cv2.findContours(
          thresh, cv2.RETR_EXTERNAL,
          cv2.CHAIN_APPROX_SIMPLE) 
       max_cntr = max(contours, key=lambda x: cv2.contourArea(x))
       epsilon = 0.005 * cv2.arcLength(
            max_cntr, True
       )  # maximum distance from contour to approximated contour.
       max_cntr = cv2.approxPolyDP(max_cntr, epsilon, True)

  • Centroid of contour of hand is found out which will be used as a center of mass for tracking hand motion.

       def centroid(contour):
          if len(contour) == 0:  # if the array is empty return (-1,-1)
             return (-1, -1)
          M = cv2.moments(
             contour)  # gives a dictionary of all moment values calculated
          try:
              x = int(
               M['m10'] / M['m00']
              )  # Centroid is given by the relations, 𝐶𝑥 =𝑀10/𝑀00 and 𝐶𝑦 =𝑀01/𝑀00
              y = int(M['m01'] / M['m00'])
          except ZeroDivisionError:
              return (-1, -1)
          return (x, y)

  • Hand motion is tracked by using the coordinates of centroid in different frames.

    Let say there are two frames: F1 and F2. The position of centroid in F1 is (x1,y1) and in F2 is (x2,y2). The time difference b/w F1 and F2 is t. Then hand motion can be tracked as shown below:

       def velocity(x1, x2, t):
           return (x2 - x1) / t
    
       def detect_motion(x1, y1, x2, y2, t):
           vel_x = velocity(x1, x2, t)
           vel_y = velocity(y1, y2, t)
    
           if vel_x > VEL_THRESHOLD:
               return MOTION_RIGHT
           elif vel_x < -VEL_THRESHOLD:
               return MOTION_LEFT
           elif vel_y > VEL_THRESHOLD:
               return MOTION_DOWN
           elif vel_y < -VEL_THRESHOLD:
               return MOTION_UP
           else:
               return NO_MOTION

  • Hand motion is mapped to keyboard's keys using PyAutoGUI for controlling the media player.

       def control_media_player(hand_motion):
           if hand_motion == MOTION_RIGHT:
               pyautogui.press(KEY_FORWARD)
           elif hand_motion == MOTION_LEFT:
               pyautogui.press(KEY_BACKWARD)
           elif hand_motion == MOTION_UP:
               pyautogui.press(KEY_VOLUME_UP)
           elif hand_motion == MOTION_DOWN:
               pyautogui.press(KEY_VOLUME_DOWN)

• Eye Detection

  Eye detection is done using mediapipe for tracking the user's eyes. If the user is not looking at the screen, then the media player pauses and when user looked back at the screen, it resumes.

     eyes = EyeDetection.detectEyes(frame) > 0
     if eyes and not IS_VIDEO_PLAYING:
        play_video()
     elif not eyes and IS_VIDEO_PLAYING:
        pause_video()

Demonstrations

Coming soon...

Contributions

Members of EmR are requested to refer to the M_CONTRIBUTING.md file before raising PRs or issues. Individuals outside of EmR must refer to the CONTRIBUTING.md file before raising PRs or issues.

gesturecontrol's Issues

Video acquisition from the user's webcam

Capture the video frames from the user's desktop webcam

Testing installation

In this issue all we have to do is to try installing the repo and running it on different platforms i.e.

• Windows 11
• Windows 10
• macOS
• Linux

If anyone encounters any issues, mention it as a comment. If someone discovers a solution do the same.

First issue - hand gesture recognition

This issues explains how to raise future issues. Observe a few things:

• If your issue is a part of hand gesture recognition add a milestone tag (as added in this issue). Do not add any milestone for issues that address both parts of the project.
• Add label of hand gesture (and eye tracking if issues addresses both parts).
• Mention to whom you want to assign the issue.

This first issue will be assigned to all members in the hand gesture recognition team.

Add docstrings for all the functions in gesture_control file

User's hand contours and centroid

• Draw the contours for the user's hand using the mask created based on the user's hand color.
• Also, locate the centroid of the hand.

HSV trackbars to create a mask for the user's hand

Write the code for trackbars in a separate python file to change the lower and upper range of the HSV that would create a binary mask corresponding to the user's hand color.

Control the media player based on the state of the user's eye

• Detect the presence of the user's eyes in the video frame.
• Develop the logic to decide when to pause or play the video based on the above information.
• Control the media player based on the above logic.

You can try using the different available options like haarcascade, mediapipe, etc whichever you feel is accurate and convenient.
If you can add some element of DIP here to make this system better, that would just be awesome 😉

Control the media player based on the commands

Execute the commands derived from the user's gesture to control the media player (pause/play, volume up/down, etc).
(It may be done using pyautogui or any similar library).

Track centroid motion to give different commands

Track the motion of the centroid of the user's hand and use that information to give different commands to the media player. SOme of the commands are: -

• Forward/ back
• Volume up/down
• Play/ Pause

Control the media player based on the state of the user's eye

• Detect the presence of the user's eyes in the video frame.
• Develop the logic to decide when to pause or play the video based on the above information.
• Control the media player based on the above logic.

You can try using the different available options like haarcascade, mediapipe, etc whichever you feel is accurate and convenient.
If you can add some element of DIP here to make this system better, that would just be awesome 😉

First issue - eye/face tracking

This issues explains how to raise future issues. Observe a few things:

• If your issue is a part of eye tracking add a milestone tag (as added in this issue). Similarly do this if it is part of hand gesture recognition (you can only add a single milestone). Do not add any milestone for issues that address both parts of the project.
• Add label of eye tracking (and hand gesture if issues addresses both parts).
• Mention to whom you want to assign the issue.

This first issue will be assigned to all members in the eye/face tracking team.