pauldevos / football_player_recognition

Football Player Recognition

This project will identify the jersey numbers of American football players in broadcast footage using a two stage approach. The first stage will be a pre-trained Mask R-CNN that will detect players, and the second stage will be a fine-tuned Faster R-CNN to extract digits from the player bounding boxes.

Usage

# From repository root directory
python main/main.py <INPUT_VIDEO_PATH>

Football Dataset

1. Capture frames from two Alabama football games (1280 x 720 resolution), one with home uniforms and the other with away uniforms.

2. Do a pass over the frames, removing ones that do not contain at least one Alabama player with a completely visible jersey number.

3. Run a pre-trained Mask R-CNN to extract person bounding boxes from the frames.

4. Pad the bounding boxes with 0s (blackness) to create square images then re-scale to size 256 x 256. Discard non-Alabama players.

5. Label the digits with VGG Image Annotator (VIA) tool.

For more details, see data folder, which holds its own README.md file.

Training Approach

1. Start with the same pre-trained Mask R-CNN as above. As shown in this tutorial, it is possible to train the Mask R-CNN for object detection using a dataset that only contains bounding boxes.

2. Evaluate performance on football_player_test for the following three scenarios:

   1. Fine-tuning with Street View House Numbers (SVHN) dataset
   2. Fine-tuning with football_player_train
   3. Fine-tuning with SVHN + football_player_train

3. The best performing model is hereby referred to as jersey_number_detector.

Next Steps

1. Improve speed by pipelining the two stages
2. Synthesize the discrete digits into actual jersey numbers (i.e. '2' and '9' --> '29')
3. Filter out away team jersey numbers based on color, which will vary per game
4. Filter out sideline player noise
5. Supplement the jersey numbers with roster information (player names)
6. Add game context using OCR on the scoreboard