about target_perframe file about miniroad

[jbistanbul]

To generate target npy files, you need 1) the length of the video feature, 2) annotation files (which are annotated in seconds), and 3) the frame rate by which the video was extracted.

For convenience, let's take an example of a standard THUMOS processing pipeline using TSN. The video is processed in 24 FPS, the chunk size is 6 with no overlapping segments (equivalent to window size 6 and stride 6).

The length of the video feature can be known when extracting RGB features. Since chunk size is set as 6, the middle frame out of every 6 frames (typically using the 4th frame) is given as input to TSN, and a feature corresponding to that time segment is extracted. This process is continued throughout the whole video and it will generate a feature numpy file of length x (which is the number of frames // 6 in this case). In short, x = # frames // chunk size, ignoring the last chunk if it does not have 6 frames and assuming non-overlapping segments.

Knowing the length of the video feature x and the annotation file, you can start generating target numpy file. Read the feature numpy file to get x, create dummy zero-filled numpy array of (x, c) where c is the number of classes+1 (for background), and then, convert the annotation files to feature level simply by multiplying the annotation in seconds by FPS (in which the video was extracted) and fill these values into the numpy array. Typically, these multiplied values are not integers so we floor the start of the action and ceil the end of the action so that the feature target annotation is more lenient on segments containing actions. Lastly, for segments that are not labeled (which is the background), you need to set them to 1. For example, if mask are the indices of the background, target_npy_file[mask,0] = 1, assuming index 0 is the index for the background.

[Chenhongchang]

Thanks for your detailed response!