I get this error when running object removal on my own files :

Calculating backward flow 69 <---> 70
Finish flow prediction.
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 464, in video_completion_seamless
videoFlowF = complete_flow(args, corrFlowF, flow_mask, 'forward', FlowF_edge)
File "video_completion.py", line 194, in complete_flow
flow_mask_img = flow_mask[:, :, i] if mode == 'forward' else flow_mask[:, :, i + 1]
IndexError: index 16 is out of bounds for axis 2 with size 16

I used wget.exe version 1.21.1 (latest) on Windows.

PS C:\Users\marc-\Downloads> .\wget.exe https://filebox.ece.vt.edu/~chengao/FGVC/data.zip
--2021-08-16 20:39:46--  https://filebox.ece.vt.edu/~chengao/FGVC/data.zip
Resolving filebox.ece.vt.edu (filebox.ece.vt.edu)... 128.173.88.43
Connecting to filebox.ece.vt.edu (filebox.ece.vt.edu)|128.173.88.43|:443... failed: Unknown error.
Retrying.

Anyone had this issue?

Like the title says; it doesn't create these folders after processing. I would love to hear what I missed! :)

am I asking too much?
I'm trying to execute a sequence of 228 fullhd jpg files on Colab.
I get this message:

tcmalloc: large alloc 5673369600 bytes == 0x55b4a2bdc000 @  0x7fa98570bb6b 0x7fa98572b379 0x7fa922e4892e 0x7fa922e4a946 0x7fa95b226bb3 0x7fa95acdb17a 0x7fa95acdc293 0x7fa95b1dcedf 0x7fa95ad6a6a6 0x7fa95ad6b7b0 0x7fa95afc21c9 0x7fa95a963689 0x7fa95b0a1919 0x7fa95ad6d09b 0x7fa95b157a08 0x7fa95a963689 0x7fa95b0a1709 0x7fa95ad62d1d 0x7fa95b161dc8 0x7fa95a963689 0x7fa95b0a1b29 0x7fa95cddb347 0x7fa95a963689 0x7fa95b0a1b29 0x7fa96a73b746 0x55b2ce0770e4 0x55b2ce076de0 0x55b2ce0eb6f5 0x55b2ce07869a 0x55b2ce0e6a45 0x55b2ce07869a
bash: line 7:  1201 Killed                  python3 video_completion.py --mode object_removal --path ../kepescap/kepesc --path_mask ../kepescap/kepesc_mask --outroot ../result/kep_removal --seamless

Greetings! Thanks for sharing your impressive work.

I was trying to find the non-local flow computation and forward-backward flow consistency check in your code. But it seems that the 'NonLocal' option is disabled by default and the 'consistencyThres' is set to inf which means it won't filter out any outliers. The results still seem reasonable, though. Could you please verify if my interpretation is correct and give me some guidance on how to tune these parameters?

Cheers,
Lao

in google colab running extrapolation on the first example frame tennis has artifacts

I Have been trying to run this on colab, i have 3 frames and running the video_extrapolation mode, i cant run this on a higher resolution ex: 960x720 because CUDA runs out of memory, when i downscaled for 640x480 it gave me a new error.

Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 402, in video_completion_seamless
corrFlowF = calculate_flow(args, RAFT_model, video, 'forward')
File "video_completion.py", line 120, in calculate_flow
_, flow = model(image1, image2, iters=20, test_mode=True)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/raft.py", line 101, in forward
fmap1, fmap2 = self.fnet([image1, image2])
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/extractor.py", line 176, in forward
x = self.conv1(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 419, in forward
return self._conv_forward(input, self.weight)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 416, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[2, 4, 480, 640] to have 3 channels, but got 4 channels instead

Hi, thank you for your wonderful work!

I want to know how to obtain the mask file for a custom video?(like the mask files in ./data/tennis_mask)

just like this link

I saw 3 pre-trained models. But I don't know how to train with my own dataset.

python video_completion.py \

   --mode video_extrapolation \
   --path ../data/tennis \
   --outroot ../result/tennis_extrapolation \
   --H_scale 2 \
   --W_scale 2 \
   --seamless

Pytorch 1.6.0 for flow prediction

Finish Calculating flow.
Traceback (most recent call last):
File "video_completion.py", line 615, in
main(args)
File "video_completion.py", line 578, in main
video_completion_seamless(args)
File "video_completion.py", line 418, in video_completion_seamless
video, corrFlowF, corrFlowB, flow_mask, mask_dilated, start_point, end_point = extrapolation(args, video, corrFlowF, corrFlowB)
File "video_completion.py", line 165, in extrapolation
corrFlowF[H_start : H_start + imgH, W_start : W_start + imgW, :] = corrFlowF_ori
ValueError: could not broadcast input array from shape (512,960,2,0) into shape (512,960,2,69)

Thanks for sharing such a great job! As said in the paper, a homography warp, which is estimated using RANSAC on ORB feature matches, is used before calculating the optical flow. I'm wondering where the implementation of this operation locates since I can't find it in the code.

Colab link is not working

in your code, videoNonLocalFlowF and videoNonLocalFlowB is None?
get_flowNN_gradient(args, gradient_x_filled, gradient_y_filled, mask, mask_gradient, videoFlowF, videoFlowB, None, None)

Hi! I've problems to elaborate 500 images because I go in out of memory of ram. Moreover I tried with 2 images and It went in OOM of VRAM. Maybe resolution must be lower?

To solve #40, I compressed the images and reduced the quality. But in doing so, I am facing this error:
RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [3225, 1, 43, 75] and grid with sizes [3150, 9, 9, 2]

Hi,

Thanks for the great work! I met some problems about the cuda version. For RAFT, the requirement version of cuda is 10.1. However, for environment FGVC, cuda >10 is not supported for pytorch 0.4. It will be a littile incovenient if we need to switch the cuda version during inference.

Hi,

I am facing below error while try to run your code. The screen shot is attached.

Hi, I ran into this error, I tried to work it out myself and I found the cause but no real solution yet, I'm willing to find a reliable one and share it, if interested. Below I wrote a report detailing what I found:

I. Error and reproductibility:

Error text:

  File "e:/Work/FGVC/tool/video_completion.py", line 120, in calculate_flow
    _, flow = model(image1, image2, iters=20, test_mode=True)
  File "C:\Users\eduard\anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "e:\Work\FGVC\RAFT\raft.py", line 127, in forward
    corr = corr_fn(coords1) # index correlation volume
  File "e:\Work\FGVC\RAFT\corr.py", line 51, in __call__
    corr = bilinear_sampler(corr, coords_lvl)
  File "e:\Work\FGVC\RAFT\utils\utils.py", line 66, in bilinear_sampler
    img = F.grid_sample(img, grid, align_corners=True)
  File "C:\Users\eduard\anaconda3\envs\torch\lib\site-packages\torch\nn\functional.py", line 3390, in grid_sample
    return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [2850, 1, 38, 75] and grid with sizes [2775, 9, 9, 2]

Run into error while running a cloned code from https://github.com/vt-vl-lab/FGVC Version 1.0 with the following command

video_completion.py --mode object_removal --path E:/Work/video_segmentation/images_/ --path_mask E:/Work/video_segmentation/masks_/ --outroot ../result

On a set of 10 images/masks pairs, with the shapes [3, 300, 600]

Error origin:

The functiongrid_sampler(...) is used by bilinear_sampler(...) having the arguments corr and coords_lvl with different shapes [2850, 1, 38, 75] and [2775, 9, 9, 2]

corr = bilinear_sampler(corr, coords_lvl)

which breaks grid_sampler(...) because of the different dimmensions

In the next 3 chapters I follow both objects trail to find why they have different shapes

I. coords_lvl : CorrBlock.call(self, coords) -> cendroid_lvl

In raft.py the initialize_flow() function computes the size of the grid as being (1, 37, 75) from image with shape (1, 300, 600) because H/8 = 37.5 and H//8 = 37

    def initialize_flow(self, img):
        """ Flow is represented as difference between two coordinate grids flow = coords1 - coords0"""
        N, C, H, W = img.shape

        coords0 = coords_grid(N, H//8, W//8).to(img.device)
        coords1 = coords_grid(N, H//8, W//8).to(img.device)

        # optical flow computed as difference: flow = coords1 - coords0
        return coords0, coords1

This propgates to corr_fn() -> CorrBlock.call() that receive the coordinates as being : torch.Size([1, 2, 37, 75])

        coords0, coords1 = self.initialize_flow(image1)
        #...
        corr = corr_fn(coords1) # index correlation volume

        corr = CorrBlock.corr(fmap1, fmap2)
        batch, h1, w1, dim, h2, w2 = corr.shape
        corr = corr.reshape(batch*h1*w1, dim, h2, w2)

        self.corr_pyramid.append(corr)
        for i in range(self.num_levels-1):
            corr = F.avg_pool2d(corr, 2, stride=2)
            self.corr_pyramid.append(corr)

This is used to reshape centroid_lvl in method call(...) to [2775, 1, 1, 2]), which is then used in bilinear_sampler() with its shape being torch.Size([2775, 1, 1, 2]), which in turn gives the shape to coords_lvl

II. corr <- fmaps:

corr eventually takes its shape from fmaps, as detailed in ch.III

Fmaps are generated (in this case) using a BasicEncoder(nn.Module). Looking at the forward(self, x) method,

    def forward(self, x):

        # if input is list, combine batch dimension
        is_list = isinstance(x, tuple) or isinstance(x, list)
        if is_list:
            batch_dim = x[0].shape[0]
            x = torch.cat(x, dim=0)

        x = self.conv1(x)
        x = self.norm1(x)
        x = self.relu1(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)

        x = self.conv2(x)

        if self.training and self.dropout is not None:
            x = self.dropout(x)

        if is_list:
            x = torch.split(x, [batch_dim, batch_dim], dim=0)
        return x

It returns a feature map from an image, by passing it through different nn layers, with its shape at the end being exactly torch.Size([256, 38, 75]). This shape is propagates as described in chapter III.

III. fmap1, fmap2 -> CorrBlock.init(self, fmap1, fmap2, num_levels=4, radius=4) -> corr.shape

In raft.py, method self.fnet([image1, image2]) return fmap1 and fmap2, with shapes torch.Size([1, 256, 38, 75])

    with autocast(enabled=self.args.mixed_precision):
        fmap1, fmap2 = self.fnet([image1, image2])

This propagates to CorrBlock.init() to corr object with shape
torch.Size([2850, 1, 38, 75]) where 2850=38*75

It is then appended to self.corr_pyramid, to finally be used in the call(), in bilinear_sampler() with its shape being torch.Size([2850, 1, 1, 2])

IV. Possible solutions:

To match the same shape, either the small CNN must be modified, or the way the grid shape its defined in initialize_flow(), from:

coords0 = coords_grid(N, H//8, W//8).to(img.device)

to:

coords0 = coords_grid(N, np.round(H/8), np.round(W/8))).to(img.device)

However, I suspect this change should be made at other points in the implementation as well.

V. Observation

This method of dividing by 8 to match the output shape of the convolutions can raise multiple errors, and should better match exactly the output shape. In case of modifying the architecture of the CNN, this will also throw shape miss-match errors.

I'm using Colab to run this.

Whenever I use the video completion, the video output is very skippy. A couple of frames will play nicely, but then the video skips to a couple frames back and then quickly skips back. I can tell the video is being extrapolated, but this skipping makes playback not smooth.

Can this technology apply to remove the real-time video? on-line?

RuntimeError: CUDA out of memory. Tried to allocate 3.52 GiB (GPU 0; 15.75 GiB total capacity; 11.38 GiB already allocated; 1.07 GiB free; 11.54 GiB reserved in total by PyTorch)
(Running on Colab Pro)
What is the minimum required specs to run the current version of FGVC ?

can you export the pth for version1.4.0? I cant load the .pth file.

Hi all, sorry for the inconvenience.

Please check the latest ver1.0 code. You only need to set up one environment, i.e., Pytorch 1.6.0, CUDA 10.1. There is no need to switch between different environments.

Please let me know if you have any questions. Thanks.

Issue: Program still asks me to switch to Pytorch 0.4.0 after I've switched to FGVC environment for the 2nd execution which has torch 0.4.0 installed

I've setup the environments as asked by the referenced githubs. Then I've tried the object removal example from the quickstart:

python video_completion.py \
       --mode object_removal \
       --path ../data/tennis \
       --path_mask ../data/tennis_mask \
       --outroot ../result/tennis_removal \
       --seamless

which also worked almost flawlessly in the raft environment. It created the result folder while also filling the subfolders with segmented images that look right (for example:
).

After changing the environment to FGVC & executing it again it still asks me to switch to pytorch 0.4.0 (-> change environment to FGVC) after executing it.

conda list torch indicates I've indeed installed the correct torch version being 0.4.0:

"conda list torch" produces:

(FGVC) D:\Weltherrschaft\AI\FGVC\FGVC-master\tool>conda list torch
# packages in environment at D:\Programme\Anaconda\Anaconda\envs\FGVC:
#
# Name                    Version                   Build  Channel
torch                     0.4.0                    pypi_0    pypi
torchvision               0.2.1                    pypi_0    pypi

I've also tried pytorch 0.4.1, same issue. Even though the github suggests Linux I've ran this on Windows 10. I'm assuming this cannot be the root of the issue because everything else seemingly worked well.

Image of relevant CMD output:

PS: For every German: Don't mind the "Weltherrschaft" ;-)

(Edit) Solution: zxyjfj's answer provides a step by step guide which, considering the reactions, seems to fix the issue for everyone.
I fixed it myself by locating the print in the source code & removing the try/catch around it so it would direct me to where the code fails.
Turns out it didn't have anything to do with the pytorch version.
They tried to import something that didn't exist in the directory & was no package and it wasn't even needed.
At another point they used an absolute path which obviously won't work unless your directories are called exactly like theirs.

I first tried with your example and everything executes fine but when it comes to retrieving the output I get this message:

unable to retrieve /result/tennis_removal/frame_seamless_comp_final/ from backend

After generating a sequence of images, gif and the mp4 file I'm getting the following error on Centos 7.

Any ideas?

Traceback (most recent call last):
File "video_completion.py", line 615, in
main(args)
File "video_completion.py", line 578, in main
video_completion_seamless(args)
File "video_completion.py", line 558, in video_completion_seamless
mask, video_comp = spatial_inpaint(deepfill, mask, video_comp)
File "/home/megatron/FGVC/tool/spatial_inpaint.py", line 12, in spatial_inpaint
img_res = deepfill.forward(video_comp[:, :, :, keyFrameInd] * 255., mask[:, :, keyFrameInd]) / 255.
File "/home/megatron/FGVC/tool/frame_inpaint.py", line 35, in forward
_, inpaint_res, _ = self.deepfill(image.to(self.device), mask.to(self.device), small_mask.to(self.device))
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/DeepFill.py", line 25, in forward
stage1_output, resized_mask = self.stage_1(stage1_input, mask)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/DeepFill.py", line 58, in forward
x = self.down(x)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/ops.py", line 116, in forward
return self.out(x)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/ops.py", line 52, in forward
x = self.conv(x)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

I saw a parameters in argparse called 'alpha', What does the params alpha stand for?

Hi there,

I was able to run the test (tennis) clips, but when I try it out on my own material I get the following error. I've tried it with 4K and HD material and I keep getting the same result:

miniconda3/lib/python3.8/site-packages/torch/nn/functional.py:3384: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 578, in main
video_completion(args)
File "video_completion.py", line 368, in video_completion
mask_tofill, video_comp = spatial_inpaint(deepfill, mask_tofill, video_comp)
File "/lgNet/users/andy/Downloads/FGVC-master/tool/spatial_inpaint.py", line 12, in spatial_inpaint
img_res = deepfill.forward(video_comp[:, :, :, keyFrameInd] * 255., mask[:, :, keyFrameInd]) / 255.
File "/lgNet/users/andy/Downloads/FGVC-master/tool/frame_inpaint.py", line 35, in forward
_, inpaint_res, _ = self.deepfill(image.to(self.device), mask.to(self.device), small_mask.to(self.device))
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/lgNet/users/andy/Downloads/FGVC-master/models/DeepFill_Models/DeepFill.py", line 27, in forward
stage2_output, offset_flow = self.stage_2(stage2_input, small_mask)
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/lgNet/users/andy/Downloads/FGVC-master/models/DeepFill_Models/DeepFill.py", line 78, in forward
attn_x, offset_flow = self.CAttn(attn_x, attn_x, mask=resized_mask)
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(input, **kwargs)
File "/lgNet/users/andy/Downloads/FGVC-master/models/DeepFill_Models/ops.py", line 343, in forward
yi = F.softmax(yiscale, dim=1)
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 1498, in softmax
ret = input.softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 3.91 GiB (GPU 0; 23.88 GiB total capacity; 16.52 GiB already allocated; 2.16 GiB free; 20.72 GiB reserved in total by PyTorch)

I install both python environments, and can successfully run the two projects.
Then I clone FGVC, and follows the instructions, it works well for the first step for raft with torch=1.6.0, but when I switch to FGVC (with torch=0.4.0), the torch is still raft, and run 1.6.0.
However, I have 0.4.0 in the pip list.
I tried to restart a new terminal, it still not work.
The python run with torch=1.6.0，I paste a screenshot below.

It seems that, no matter which python environments I use, the python script will change to torch=1.6.0

What would the mask input be? An alpha mask where white is the object to be removed?

Just wanted to ask if you are planning on simplifying the prerequisites by porting to only one version of pyTorch (tentative Nov.)? If that’s not an option maybe containerize the components to make them easier to use?
Impressive results btw. congratulations! And thanks for sharing your work!

Thank you for this great project! My GPU is 2080ti with CUDA10.2. Therefore I'm wondering if it is gonna support windows os somehow in the future?

Thank you again!

Trying

%%shell
cd FGVC
cd tool
python /content/FGVC/tool/video_completion.py \
       --mode video_extrapolation \
       --path /content/FGVC/data/tennis \
       --outroot /content/FGVC/result/tennis_extrapolation \
       --H_scale 2 \
       --W_scale 2 \
       #--seamless

in Google Colab returns:

Pytorch 1.6.0 for flow prediction

Finish Calculating flow.
tcmalloc: large alloc 1651507200 bytes == 0x117076000 @  0x7fb727903001 0x7fb71bdac765 0x7fb71be10bb0 0x7fb71be12a4f 0x7fb71bea9048 0x50a7f5 0x50cfd6 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x50b053 0x634dd2 0x634e87 0x63863f 0x6391e1 0x4b0dc0 0x7fb7274feb97 0x5b26fa
Completing forward flow  0 <--->  1
tcmalloc: large alloc 1767358464 bytes == 0x7fb589f70000 @  0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c1395753 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
tcmalloc: large alloc 1767358464 bytes == 0x7fb5209f4000 @  0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c139576a 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
tcmalloc: large alloc 1767358464 bytes == 0x7fb4cb160000 @  0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c1395753 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
tcmalloc: large alloc 1767358464 bytes == 0x7fb5346dc000 @  0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c139576a 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
Completing forward flow  1 <--->  2
Completing forward flow  2 <--->  3
Completing forward flow  3 <--->  4
Completing forward flow  4 <--->  5
Completing forward flow  5 <--->  6
Completing forward flow  6 <--->  7
Completing forward flow  7 <--->  8
Completing forward flow  8 <--->  9
Completing forward flow  9 <---> 10
Completing forward flow 10 <---> 11
Completing forward flow 11 <---> 12
Completing forward flow 12 <---> 13
Completing forward flow 13 <---> 14
Completing forward flow 14 <---> 15
Completing forward flow 15 <---> 16
Completing forward flow 16 <---> 17
Completing forward flow 17 <---> 18
Completing forward flow 18 <---> 19
Completing forward flow 19 <---> 20
Completing forward flow 20 <---> 21
Completing forward flow 21 <---> 22
Completing forward flow 22 <---> 23
Completing forward flow 23 <---> 24
Completing forward flow 24 <---> 25
Completing forward flow 25 <---> 26
Completing forward flow 26 <---> 27
Completing forward flow 27 <---> 28
Completing forward flow 28 <---> 29
Completing forward flow 29 <---> 30
Completing forward flow 30 <---> 31
Completing forward flow 31 <---> 32
Completing forward flow 32 <---> 33
Completing forward flow 33 <---> 34
Completing forward flow 34 <---> 35
Completing forward flow 35 <---> 36
Completing forward flow 36 <---> 37
Completing forward flow 37 <---> 38
Completing forward flow 38 <---> 39
Completing forward flow 39 <---> 40
Completing forward flow 40 <---> 41
Completing forward flow 41 <---> 42
Completing forward flow 42 <---> 43
Completing forward flow 43 <---> 44
Completing forward flow 44 <---> 45
Completing forward flow 45 <---> 46
Completing forward flow 46 <---> 47
Completing forward flow 47 <---> 48
Completing forward flow 48 <---> 49
Completing forward flow 49 <---> 50
Completing forward flow 50 <---> 51
Completing forward flow 51 <---> 52
Completing forward flow 52 <---> 53
Completing forward flow 53 <---> 54
Completing forward flow 54 <---> 55
Completing forward flow 55 <---> 56
Completing forward flow 56 <---> 57
Completing forward flow 57 <---> 58
Completing forward flow 58 <---> 59
Completing forward flow 59 <---> 60
Completing forward flow 60 <---> 61
Completing forward flow 61 <---> 62
Completing forward flow 62 <---> 63
Completing forward flow 63 <---> 64
Completing forward flow 64 <---> 65
Completing forward flow 65 <---> 66
Completing forward flow 66 <---> 67
Completing forward flow 67 <---> 68
Completing forward flow 68 <---> 69
Completing backward flow  0 <--->  1
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-2-4abcfdc683a5> in <module>()
----> 1 get_ipython().run_cell_magic('shell', '', 'cd FGVC\ncd tool\npython /content/FGVC/tool/video_completion.py \\\n       --mode video_extrapolation \\\n       --path /content/FGVC/data/tennis \\\n       --outroot /content/FGVC/result/tennis_extrapolation \\\n       --H_scale 2 \\\n       --W_scale 2 \\\n       #--seamless')

2 frames
/usr/local/lib/python3.6/dist-packages/google/colab/_system_commands.py in check_returncode(self)
    136     if self.returncode:
    137       raise subprocess.CalledProcessError(
--> 138           returncode=self.returncode, cmd=self.args, output=self.output)
    139 
    140   def _repr_pretty_(self, p, cycle):  # pylint:disable=unused-argument

CalledProcessError: Command 'cd FGVC
cd tool
python /content/FGVC/tool/video_completion.py \
       --mode video_extrapolation \
       --path /content/FGVC/data/tennis \
       --outroot /content/FGVC/result/tennis_extrapolation \
       --H_scale 2 \
       --W_scale 2 \
       #--seamless' died with <Signals.SIGKILL: 9>.

Thankyou for the great code.

"Results in the object removal setting, however, are difficult to compare and evaluate due to the lack of ground truth content behind the masked object. For this reason, we introduce a further synthetic object mask inpainting task. Specifically, we take a collection of free-form object masks and randomly pair them with other videos, pretending there is an object occluding the scene"

Can you please explain it with concrete example how you are evaluating your code on object removal task. Are you using some other dataset for which ground truth is available. I wanted to compare results for my project and I am not able to get the evaluation(PSNR, SSIM, LPIPS) and training code. Can you point me to some public library for this, it would be very useful.

Thanks for sharing your impressive work!
I was trying to find the non-local flow computation, but how to get these parameters of videoNonLocalFlowF and videoNonLocalFlowB in get_flowNN.py? Could you please give me some guidance for these parameters?

I'm running it on Google Colab pro (Max ram settings)
Processing 256 frames at 640x480

Calculating backward flow 146 <---> 147
Finish flow prediction.
Completing backward flow 118 <---> 119tcmalloc: large alloc 1091174400 bytes == 0xf3156000 @ 0x7fe0f8733001 0x7fe0eda164ff 0x7fe0eda66ab8 0x7fe0eda6abb7 0x7fe0edb09003 0x50a4a5 0x50cc96 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281 0x4b0dc0 0x7fe0f832ebf7 0x5b259a
tcmalloc: large alloc 1639604224 bytes == 0x7fdf0c268000 @ 0x7fe0f87311e7 0x7fe0eda1641e 0x7fe0eda66bdb 0x7fe0eda66c78 0x7fe0edab7c39 0x7fe0edb05f70 0x7fe0edb06a6b 0x566fae 0x59fd0e 0x7fe0eda53ea7 0x50a12f 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281
tcmalloc: large alloc 1639604224 bytes == 0x7fde69622000 @ 0x7fe0f87311e7 0x7fe0eda1641e 0x7fe0eda66bdb 0x7fe0eda66c78 0x7fe0edab7c39 0x7fe0edb05f70 0x7fe0edb06a6b 0x566fae 0x59fd0e 0x7fe0eda53ea7 0x50a12f 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281
bash: line 7: 325 Killed python3 video_completion.py --mode video_extrapolation --path ../demo/papa --outroot ../result/tennis_extrapolation --W_scale 2 --seamless

I am using some different images and getting this error on Colab. I would be grateful to receive some advice on this. Thanks!

I am having almost zero % GPU load when running the same code from colab. Any ideas what might be causing it?

Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 510, in video_completion_seamless
None)
File "/content/FGVC/tool/get_flowNN_gradient.py", line 440, in get_flowNN_gradient
videoNonLocalFlowB[:, :, :, 0, indFrame],
TypeError: 'NoneType' object is not subscriptable

Hi there,

Thanks for the tool, it's pretty awesome! I noticed that *.core intermediary files in FGVC/tool/ do not get cleanup up. The network seems to produce one every so often and I realized that that was happening after my hard drive filled with 300ish GB of these files. Anyways, just letting you know.

Thanks,

Leo

Literally just keeps saying this:

Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 402, in video_completion_seamless
corrFlowF = calculate_flow(args, RAFT_model, video, 'forward')
File "video_completion.py", line 120, in calculate_flow
_, flow = model(image1, image2, iters=20, test_mode=True)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/raft.py", line 101, in forward
fmap1, fmap2 = self.fnet([image1, image2])
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/extractor.py", line 176, in forward
x = self.conv1(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 399, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[2, 4, 72, 128] to have 3 channels, but got 4 channels instead

Hi! Can I use it directly with a video file? Putting the corresponding mask video on another folder? Or i need single frames?

I tried with single frames but seems that ram finishes very fast. With colab pro is it usable? In general this application can manage folder with 2600 files?

Hi,

May I ask, why this line exists ?

and this line

These will make the saved images to have green masks on top.