open-mmlab / mmflow Goto Github PK
View Code? Open in Web Editor NEWOpenMMLab optical flow toolbox and benchmark
Home Page: https://mmflow.readthedocs.io/en/latest/
License: Apache License 2.0
OpenMMLab optical flow toolbox and benchmark
Home Page: https://mmflow.readthedocs.io/en/latest/
License: Apache License 2.0
I was a member of OpenMMLab and published an optical flow estimation model CSFlow (paper: https://arxiv.org/abs/2202.00909). The work was received by IV 2022 (2022 33rd IEEE Intelligent Vehicles Symposium). I want to add this model to MMFlow.
I am not familiar with the workflow to contibute a model. Could you check if my to-do list is complete ? Can I create a PR directly so that we can discuss on the PR?
My to-do list:
Thank you!
Thanks for your great work!
Test input script:
python tools/test.py configs/raft/raft_8x2_100k_flyingchairs_368x496.py /ckpt/raft_8x2_100k_flyingchairs.pth --eval EPE
The error message is as follow:
[ ] 0/640, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/test.py", line 178, in
main()
File "tools/test.py", line 171, in main
f'In {dataset_name} '
File "root/framework/optical_flow/mmflow/mmflow/core/evaluation/evaluation.py", line 38, in online_evaluation
model, data_loader, metric=metric, **kwargs)
File "root/framework/optical_flow/mmflow/mmflow/core/evaluation/evaluation.py", line 68, in single_gpu_online_evaluation
batch_results = model(test_mode=True, **data)
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 48, in forward
return self.module(*inputs[0], **kwargs[0])
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "root/framework/optical_flow/mmflow/mmflow/models/flow_estimators/base.py", line 61, in forward
return self.forward_test(*args, **kwargs)
File "root/framework/optical_flow/mmflow/mmflow/models/flow_estimators/raft.py", line 145, in forward_test
feat1, feat2, h_feat, cxt_feat = self.extract_feat(imgs)
File "root/framework/optical_flow/mmflow/mmflow/models/flow_estimators/raft.py", line 72, in extract_feat
feat1 = self.encoder(img1)
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "root/framework/optical_flow/mmflow/mmflow/models/encoders/raft_encoder.py", line 293, in forward
x = self.conv1(x)
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 443, in forward
return self._conv_forward(input, self.weight, self.bias)
File "root/miniconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 440, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 5-dimensional input of size [1, 1, 6, 384, 512] instead
May you help me on this?
Describe the bug
The hyperlink in intro.md that should open the "get_started" page is not valid
Reproduction
Bug fix
In intro.md file, change the hyperlink from get_started to https://github.com/open-mmlab/mmflow/blob/master/docs/en/get_started.md
Great project, i'm going to study how to use it
Hi! Thank you for your good work!
Will this work support the use of C++? If not, could you give me some advice about the deployment method for the C++ project?
Thanks for your great work!
I am new to optical flow. Now I want to use occlusion_estimation()
defined in occlusion_estimation.py
. I see the annotations "flow_fw (Tensor): The forward flow with shape (N, 2, H, W)". Since now I get 25 extracted frames each video, I am wondering how to calculate the forward flow and backward flow? Do I need to iteratively rum image_demo.py
and then concatenate all the results on the first dimension?
Thanks again for your great work.
mmflow/configs/pwcnet/metafile.yml
Line 117 in 203ea55
during dataset download wegt https://lmb.informatik.uni-freiburg.de/data/FlyingChairs/FlyingChairs.zip,
remember to change wegt to wget
in mmflow/docs/dataset_prepare.md
Hi!
Bug description
I have got a problem with gma_plus-p_8x2_50k_kitti2015_288x960.pth model. Config and checkpoint was downloaded from github manually.
Reproduction
I use code from basic inference tutorial:
config_file = '/home/aa_siv/projects/flow/mmflow/configs/gma/gma_plus-p_8x2_50k_kitti2015_288x960.py'
checkpoint_file = '/home/aa_siv/projects/flow/notebooks/modelzoo_mmflow/gma_plus-p_8x2_50k_kitti2015_288x960.pth'
model = init_model(config_file, checkpoint_file, device='cuda:1')
result = inference_model(model, img1_path, img2_path)
write_flow(result, flow_file='flow_gma.flo')
flow_map = visualize_flow(result, save_file='flow_map_gma.png')
This code worked well with FlowNetC models, RAFT, but it is crushing with GMA plus p model.
It returns error:
Error traceback
RuntimeError Traceback (most recent call last)
/home/aa_siv/projects/flow/notebooks/test_opencv_flow.ipynb Cell 24' in <module>
[7](vscode-notebook-cell://ssh-remote%2B86.62.87.164/home/aa_siv/projects/flow/notebooks/test_opencv_flow.ipynb#ch0000022vscode-remote?line=6) model = init_model(config_file, checkpoint_file, device='cuda:1')
[9](vscode-notebook-cell://ssh-remote%2B86.62.87.164/home/aa_siv/projects/flow/notebooks/test_opencv_flow.ipynb#ch0000022vscode-remote?line=8) # test image pair, and save the results
---> [10](vscode-notebook-cell://ssh-remote%2B86.62.87.164/home/aa_siv/projects/flow/notebooks/test_opencv_flow.ipynb#ch0000022vscode-remote?line=9) result = inference_model(model, img1_path, img2_path)
[11](vscode-notebook-cell://ssh-remote%2B86.62.87.164/home/aa_siv/projects/flow/notebooks/test_opencv_flow.ipynb#ch0000022vscode-remote?line=10) # save the optical flow file
[12](vscode-notebook-cell://ssh-remote%2B86.62.87.164/home/aa_siv/projects/flow/notebooks/test_opencv_flow.ipynb#ch0000022vscode-remote?line=11) write_flow(result, flow_file='flow_gma.flo')
File ~/projects/flow/mmflow/mmflow/apis/inference.py:151, in inference_model(model, img1s, img2s, valids)
[149](file:///home/aa_siv/projects/flow/mmflow/mmflow/apis/inference.py?line=148) # forward the model
[150](file:///home/aa_siv/projects/flow/mmflow/mmflow/apis/inference.py?line=149) with torch.no_grad():
--> [151](file:///home/aa_siv/projects/flow/mmflow/mmflow/apis/inference.py?line=150) results = model(test_mode=True, **data)
[153](file:///home/aa_siv/projects/flow/mmflow/mmflow/apis/inference.py?line=152) if valid_masks[0] is not None:
[154](file:///home/aa_siv/projects/flow/mmflow/mmflow/apis/inference.py?line=153) # filter the output flow map
[155](file:///home/aa_siv/projects/flow/mmflow/mmflow/apis/inference.py?line=154) for result, valid in zip(results, valid_masks):
File ~/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
[1098](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
[1099](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
[1100](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
[1101](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1100) or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1101) return forward_call(*input, **kwargs)
[1103](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
[1104](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []
File ~/projects/flow/mmflow/mmflow/models/flow_estimators/base.py:61, in FlowEstimator.forward(self, test_mode, *args, **kwargs)
[59](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/base.py?line=58) return self.forward_train(*args, **kwargs)
[60](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/base.py?line=59) else:
---> [61](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/base.py?line=60) return self.forward_test(*args, **kwargs)
File ~/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py:151, in RAFT.forward_test(self, imgs, flow_init, img_metas)
[148](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=147) if flow_init is None:
[149](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=148) flow_init = torch.zeros((B, 2, H, W), device=feat1.device)
--> [151](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=150) results = self.decoder.forward_test(
[152](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=151) feat1=feat1,
[153](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=152) feat2=feat2,
[154](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=153) flow=flow_init,
[155](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=154) h_feat=h_feat,
[156](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=155) cxt_feat=cxt_feat,
[157](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=156) img_metas=img_metas)
[158](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=157) # recover iter in train
[159](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/flow_estimators/raft.py?line=158) self.decoder.iters = train_iter
File ~/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py:504, in RAFTDecoder.forward_test(self, feat1, feat2, flow, h_feat, cxt_feat, img_metas)
[482](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=481) def forward_test(self,
[483](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=482) feat1: torch.Tensor,
[484](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=483) feat2: torch.Tensor,
(...)
[487](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=486) cxt_feat: torch.Tensor,
[488](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=487) img_metas=None) -> Sequence[Dict[str, np.ndarray]]:
[489](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=488) """Forward function when model training.
[490](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=489)
[491](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=490) Args:
(...)
[502](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=501) with the same size of images before augmentation.
[503](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=502) """
--> [504](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=503) flow_pred = self.forward(feat1, feat2, flow, h_feat, cxt_feat)
[506](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=505) flow_result = flow_pred[-1]
[507](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/raft_decoder.py?line=506) # flow maps with the shape [H, W, 2]
File ~/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py:283, in GMADecoder.forward(self, feat1, feat2, flow, h, cxt_feat)
[280](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=279) upflow_preds = []
[281](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=280) delta_flow = torch.zeros_like(flow)
--> [283](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=282) attention = self.attn(cxt_feat)
[285](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=284) for _ in range(self.iters):
[286](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=285) flow = flow.detach()
File ~/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
[1098](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
[1099](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
[1100](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
[1101](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1100) or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1101) return forward_call(*input, **kwargs)
[1103](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
[1104](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []
File ~/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py:135, in Attention.forward(self, x)
[132](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=131) self_similarity = torch.zeros((B, self.heads, H * W, H * W)).to(q)
[134](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=133) if self.max_pos_size is not None:
--> [135](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=134) self_similarity += self.pos_emb(q)
[137](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=136) if not self.position_only:
[138](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=137) # self_similarity shape is (B, heads, HxW, HxW)
[139](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=138) self_similarity += torch.matmul(
[140](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=139) q.view(B, self.heads, -1, self.head_channels),
[141](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=140) k.view(B, self.heads, self.head_channels, -1)) * self.scale
File ~/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py:1102, in Module._call_impl(self, *input, **kwargs)
[1098](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1097) # If we don't have any hooks, we want to skip the rest of the logic in
[1099](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1098) # this function, and just call forward.
[1100](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1099) if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
[1101](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1100) or _global_forward_hooks or _global_forward_pre_hooks):
-> [1102](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1101) return forward_call(*input, **kwargs)
[1103](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1102) # Do not call functions when jit is used
[1104](file:///home/aa_siv/projects/flow/venv/lib/python3.9/site-packages/torch/nn/modules/module.py?line=1103) full_backward_hooks, non_full_backward_hooks = [], []
File ~/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py:45, in RelPosEmb.forward(self, q)
[41](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=40) B, heads, H, W, head_channels = q.shape
[43](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=42) height_emb = self.rel_height(self.rel_ind[:H, :H].reshape(-1)).reshape(
[44](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=43) H, H, -1, head_channels)
---> [45](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=44) width_emb = self.rel_width(self.rel_ind[:W, :W].reshape(-1)).reshape(
[46](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=45) W, -1, W, head_channels)
[48](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=47) height_score = torch.einsum('b h x y d, x u v d -> b h x y u v', q,
[49](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=48) height_emb)
[50](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=49) width_score = torch.einsum('b h x y d, y u v d -> b h x y u v', q,
[51](file:///home/aa_siv/projects/flow/mmflow/mmflow/models/decoders/gma_decoder.py?line=50) width_emb)
RuntimeError: shape '[240, -1, 240, 128]' is invalid for input of size 3276800
I haven't done any code modifications.
I used 2 images with shape 1920, 1080.
Environment
torch 1.10.2+cu113
torchaudio 0.10.2+cu113
torchvision 0.11.3+cu113
mmcv-full 1.5.3
mmflow 0.4.2 /home/aa_siv/projects/flow/mmflow
PyTorch was installed via pip with command:
pip install torch==1.10.2+cu113 torchvision==0.11.3+cu113 torchaudio==0.10.2+cu113 -f https://download.pytorch.org/whl/torch_stable.html
Bug fix
I am new to optical flow and i can't help with bugfix :(
Update the zh_cn docs file 0_config_.md
.
Translate original english file into Chinese.
Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
I am new to the optical flow and have a basic question.
To learn to predict optical flow, we need the ground truth for supervision. My understanding is, for example, the TV-L1. But since the groud truth optical flow can be computed by a non-CNN algorith,, why we should use a CNN to estimate the optical flow? Is that much faster than the non-CNN method?
1, 'Ground truth length doesn't match video frames'
AssertionError: Ground truth length doesn't match video frames
I found some model cannot handle input resolution even divided by 16, e.g. 224 or 240.
An example:
import torch
from mmflow.apis import init_model
# Specify the path to model config and checkpoint file
config_file = 'configs/pwcnet_plus_8x1_750k_sintel_kitti2015_hd1k_320x768.py'
checkpoint_file = 'checkpoints/pwcnet_plus_8x1_750k_sintel_kitti2015_hd1k_320x768.pth'
# build the model from a config file and a checkpoint file
model = init_model(config_file, checkpoint_file, device='cuda:0')
with torch.no_grad():
out = model(imgs=torch.rand(1, 6, 224, 224).cuda())
Error:
Traceback (most recent call last):
File "/home/louis/PycharmProjects/mmflow/memory_and_speed.py", line 21, in <module>
out = model(imgs=torch.rand(1, 6, 224, 224).cuda())
File "/home/louis/miniconda3/envs/mmflow/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/louis/miniconda3/envs/mmflow/lib/python3.10/site-packages/mmflow/models/flow_estimators/base.py", line 61, in forward
return self.forward_test(*args, **kwargs)
File "/home/louis/miniconda3/envs/mmflow/lib/python3.10/site-packages/mmflow/models/flow_estimators/pwcnet.py", line 90, in forward_test
return self.decoder.forward_test(feat1, feat2, H, W, img_metas)
File "/home/louis/miniconda3/envs/mmflow/lib/python3.10/site-packages/mmflow/models/decoders/pwcnet_decoder.py", line 320, in forward_test
flow_pred = self.forward(feat1, feat2)
File "/home/louis/miniconda3/envs/mmflow/lib/python3.10/site-packages/mmflow/models/decoders/pwcnet_decoder.py", line 258, in forward
corr_feat = torch.cat((corr_feat_, _feat1, upflow, upfeat),
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 7 but got size 8 for tensor number 2 in the list.
Images with side of 240 and 224 result in error while 256 does not.
When I would like to finetune with other downstream task, it is not so easy to finish it on this repo? I have no good idea about that.
Hi, thanks for your great work. I am just wondering if you have any plan to support unsupervised optical flow algorithms.
There are also many works for unsueprvised optical flow, like uflow. and the performance of these algorithms are impressive considering the unsupervised learning paradigm. So i think it is a good idea to support some unsupervised optical flow algorithms.
Thanks for your error report and we appreciate it a lot.
Checklist
Describe the bug
A clear and concise description of what the bug is.
Reproduction
A placeholder for the command.
Environment
python mmdet/utils/collect_env.py
to collect necessary environment information and paste it here.$PATH
, $LD_LIBRARY_PATH
, $PYTHONPATH
, etc.)Error traceback
If applicable, paste the error trackback here.
A placeholder for trackback.
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
Thanks for your wonderful code!
I have two gpus and to reproduce the PWC-Net, I only need to set the samples_per_gpu=4
HERE and keep other settings unchanged.
And then, run
bash ./tools/dist_train.sh configs/pwcnet/pwcnet_8x1_slong_flyingchairs_384x448.py 2
.
Are these okay?
mmflow/configs/gma/metafile.yml
Line 99 in 0514e24
Hello. Some of the URLs in the model zoo, i.e.
https://github.com/open-mmlab/mmflow/blob/master/configs/gma/README.md
have spaces in that break the URL, i.e.
https://download.openmmlab.com/mmflow/gma/ gma_8x2_120k_mixed_368x768.py
instead of
https://download.openmmlab.com/mmflow/gma/gma_8x2_120k_mixed_368x768.py
Hi,
first of all cool library. :)
I wanted to know if you have seen the papers on the AutoFlow Dataset, and if support and pretrained models are planned in the near future?
Best,
Markus
import os
import sys
import argparse
import torch
from mmflow.apis import inference_model, init_model
from mmflow.datasets import visualize_flow, write_flow
def parser_func():
parser = argparse.ArgumentParser()
parser.add_argument('--config',default='configs/pwcnet/pwcnet_ft_4x1_300k_kitti_320x896.py', help='Config file')
parser.add_argument('--checkpoint', type=str, default='models/pwc/pwcnet_ft_4x1_300k_kitti_320x896.pth')
parser.add_argument('--out_path', type=str, default='models/pwc/pwcnet_ft_4x1_300k_kitti_320x896.onnx')
parser.add_argument(
'--device', default='cuda:0', help='Device used for inference')
parser.add_argument('--batch_size', type=int, default=1)
args = parser.parse_args()
os.makedirs(os.path.dirname(args.out_path), exist_ok=True)
return args
def export_onnx():
model = init_model(args.config, args.checkpoint, device=args.device)
input_names = ['encoder.layers.0.layers.0.conv']
output_names = ['decoder.decoders.level6.upfeat_layer']
dummy_input = torch.randn(args.batch_size, 6, 320, 896).cuda()
torch.onnx.export(model, dummy_input, args.out_path,
input_names=input_names, output_names=output_names, opset_version=11,
verbose=True)
if __name__ == '__main__':
args = parser_func()
export_onnx()
RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: numpy.ndarray
if I have already extracted video data into the frames folder, how to extract optical flow from the frames?
When deploy (pytorch->onnx->tensorrt) the flow model on TensorRT backend with dynamic shape(e.g., height and width), how to config the ops with parameter depending on input's shape, which is a python int
type.
Take the one below for example
mmflow/mmflow/ops/corr_lookup.py
Line 114 in 14f8d8f
Describe the bug
Good evening, i wanted to lauch train.py using a default config file for RAFT on a standard dataset (KITTI_2015).
I followed the instruction to install MMFlow from source successfully.
Reproduction
python tools/train.py configs/raft/raft_8x2_50k_kitti2015_and_Aug_288x960.py \
--load-from /home/s.starace/FlowNets/mmflow/checkpoints/raft/raft_8x2_100k_mixed_368x768.pth
Did you make any modifications on the code or config? Did you understand what you have modified?
I just changed the name of symlink that i created under /data (uppercase)
What dataset did you use?
KITTI_2015
Environment
I launched the command on my PC and also on a little cluster and the output error is the same.
Error traceback
See log attached: slurm-53090.out.txt
Bug fix
Not sure about it, could either be a configuration issue in Encoder/Decoder or a regression.
I'll try the train.py using other models as well and update the report if i understand better the problem.
Hi,
Do you have any plan to add the codes of training from scratch?
I cannot get the desired results(epe = 0.78 on flyingchairs), I check the log file provided and find that the seed is "null", so what does the null value means? Doesn't the function "init_random_seed" generate a seed?
In this repo, the correlation output is not normalized(scale=False
always), which is different from that in PWC & FlowNet repo. They normalize it by dimension. Hence, is it make little of importance?
mmflow/mmflow/models/utils/correlation_block.py
Lines 61 to 68 in 2f98716
Thanks for your error report and we appreciate it a lot.
Checklist
Describe the bug
A clear and concise description of what the bug is.
Reproduction
A placeholder for the command.
Environment
python mmdet/utils/collect_env.py
to collect necessary environment information and paste it here.$PATH
, $LD_LIBRARY_PATH
, $PYTHONPATH
, etc.)Error traceback
If applicable, paste the error trackback here.
A placeholder for trackback.
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
Good morning,
I have read the documentation about finetuning and i'd like to lauch train.py to finetune PwcNet on my dataset loading a checkpoint file.
I have created a script to load the dataset under /configs/base/datasets and used in the config script under /configs/pwcnet.
I'd like to reduce the training time, using all the resource i have available on a small cluster with 4 nodes and 4 GPU (NVLINK) NVIDIA Tesla V100 32GB SXM2.
Do you see any rooms for improvements with the following command?
srun -p xgpu --job-name=pwc_kitti --gres=gpu:4 --ntasks=16 --ntasks-per-node=4 --cpus-per-task=2 --kill-on-bad-exit=1 python -u tools/train.py $MMFLOW/configs/pwcnet/pwcnet_ft_4x1_300k_kitti_320x896.py --work-dir=$MMFLOW/work_dir/pwckitti --launcher=slurm
It estimates about 1 day to complete finetuning, do you think i'm using all 4 nodes correctly?
If so, can i reduce the training iterations to require less time?
Thanks in advance!
parser.add_argument('--config',default='configs/pwcnet/pwcnet_plus_8x1_750k_sintel_kitti2015_hd1k_320x768.py', help='Config file')
parser.add_argument('--checkpoint', default='models/pwc/pwcnet_plus_8x1_750k_sintel_kitti2015_hd1k_320x768.pth', help='Checkpoint file')
Traceback (most recent call last):
File "/home/wangzhaoming/workspace/code/mmflow/demo/video_demo.py", line 142, in
main(args)
File "/home/wangzhaoming/workspace/code/mmflow/demo/video_demo.py", line 82, in main
result = inference_model(model, img1, img2)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/apis/inference.py", line 134, in inference_model
results = model(test_mode=True, **data)
File "/disk/conda/envs/python37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/models/flow_estimators/base.py", line 61, in forward
return self.forward_test(*args, **kwargs)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/models/flow_estimators/pwcnet.py", line 90, in forward_test
return self.decoder.forward_test(feat1, feat2, H, W, img_metas)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/models/decoders/pwcnet_decoder.py", line 320, in forward_test
flow_pred = self.forward(feat1, feat2)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/models/decoders/pwcnet_decoder.py", line 259, in forward
dim=1)
RuntimeError: Sizes of tensors must match except in dimension 2. Got 24 and 23 (The offending index is 0)
Hi
My use case involves 180x360 image frames (satellite earth data). Can I configure mmflow models to take that type of input sizes?
Thanks,
Alex
运行video_demo推理demo.mp4生成demo_gt.mp4没有问题,推理自己的视频就有问题。
报错:RuntimeError: Sizes of tensors must match except in dimension 2. Got 46 and 45 (The offending index is 0)
git clone https://github.com/open-mmlab/mmflow.git
cd mmflow/
conda activate mmflow
cd checkpoints/
wget https://download.openmmlab.com/mmflow/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.pth
python demo/video_demo.py "/home/thhicv/视频/flow/2_数字_蜡烛_红外光源.mp4" configs/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.py checkpoints/pwcnet_ft_4x1_300k_sintel_final_384x768.pth demo/2_数字_蜡烛_红外光源.mp4
(mmflow) thhicv@ThinkStation-P720:~/mmflow$ python demo/video_demo.py "/home/thhicv/视频/flow/2_数字_蜡烛_红外光源.mp4" configs/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.py checkpoints/pwcnet_ft_4x1_300k_sintel_final_384x768.pth demo/2_数字_蜡烛_红外光源.mp4
Use load_from_local loader
Traceback (most recent call last):
File "demo/video_demo.py", line 141, in <module>
main(args)
File "demo/video_demo.py", line 81, in main
result = inference_model(model, img1, img2)
File "/home/thhicv/program/mmflow/mmflow/apis/inference.py", line 134, in inference_model
results = model(test_mode=True, **data)
File "/home/thhicv/anaconda3/envs/mmflow/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/thhicv/program/mmflow/mmflow/models/flow_estimators/base.py", line 61, in forward
return self.forward_test(*args, **kwargs)
File "/home/thhicv/program/mmflow/mmflow/models/flow_estimators/pwcnet.py", line 91, in forward_test
return self.decoder.forward_test(feat1, feat2, H, W, img_metas)
File "/home/thhicv/program/mmflow/mmflow/models/decoders/pwcnet_decoder.py", line 321, in forward_test
flow_pred = self.forward(feat1, feat2)
File "/home/thhicv/program/mmflow/mmflow/models/decoders/pwcnet_decoder.py", line 259, in forward
dim=1)
RuntimeError: Sizes of tensors must match except in dimension 2. Got 46 and 45 (The offending index is 0)
python tools/misc/print_cfg.py configs/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.py
dataset_type = 'Sintel'
data_root = 'data/Sintel'
img_norm_cfg = dict(
mean=[0.0, 0.0, 0.0], std=[255.0, 255.0, 255.0], to_rgb=False)
crop_size = (384, 768)
global_transform = dict(
translates=(0.05, 0.05),
zoom=(1.0, 1.5),
shear=(0.86, 1.16),
rotate=(-10.0, 10.0))
relative_transform = dict(
translates=(0.00375, 0.00375),
zoom=(0.985, 1.015),
shear=(1.0, 1.0),
rotate=(-1.0, 1.0))
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_occ=True),
dict(
type='ColorJitter',
brightness=0.5,
contrast=0.5,
saturation=0.5,
hue=0.5),
dict(type='RandomGamma', gamma_range=(0.7, 1.5)),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='RandomFlip', prob=0.5, direction='horizontal'),
dict(type='RandomFlip', prob=0.5, direction='vertical'),
dict(
type='RandomAffine',
global_transform=dict(
translates=(0.05, 0.05),
zoom=(1.0, 1.5),
shear=(0.86, 1.16),
rotate=(-10.0, 10.0)),
relative_transform=dict(
translates=(0.00375, 0.00375),
zoom=(0.985, 1.015),
shear=(1.0, 1.0),
rotate=(-1.0, 1.0))),
dict(type='RandomCrop', crop_size=(384, 768)),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['imgs', 'flow_gt'],
meta_keys=[
'img_fields', 'ann_fields', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'filename_flow',
'ori_filename_flow', 'ori_shape', 'img_shape', 'img_norm_cfg'
])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2', 'ori_filename1',
'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg',
'scale_factor', 'pad_shape'
])
]
sintel_final_train = dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_occ=True),
dict(
type='ColorJitter',
brightness=0.5,
contrast=0.5,
saturation=0.5,
hue=0.5),
dict(type='RandomGamma', gamma_range=(0.7, 1.5)),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='RandomFlip', prob=0.5, direction='horizontal'),
dict(type='RandomFlip', prob=0.5, direction='vertical'),
dict(
type='RandomAffine',
global_transform=dict(
translates=(0.05, 0.05),
zoom=(1.0, 1.5),
shear=(0.86, 1.16),
rotate=(-10.0, 10.0)),
relative_transform=dict(
translates=(0.00375, 0.00375),
zoom=(0.985, 1.015),
shear=(1.0, 1.0),
rotate=(-1.0, 1.0))),
dict(type='RandomCrop', crop_size=(384, 768)),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['imgs', 'flow_gt'],
meta_keys=[
'img_fields', 'ann_fields', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'filename_flow',
'ori_filename_flow', 'ori_shape', 'img_shape', 'img_norm_cfg'
])
],
data_root='data/Sintel',
test_mode=False,
pass_style='final')
sintel_clean_test = dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2', 'ori_filename1',
'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg',
'scale_factor', 'pad_shape'
])
],
data_root='data/Sintel',
test_mode=True,
pass_style='clean')
sintel_final_test = dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2', 'ori_filename1',
'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg',
'scale_factor', 'pad_shape'
])
],
data_root='data/Sintel',
test_mode=True,
pass_style='final')
data = dict(
train_dataloader=dict(
samples_per_gpu=1,
workers_per_gpu=5,
drop_last=True,
persistent_workers=True),
val_dataloader=dict(samples_per_gpu=1, workers_per_gpu=5, shuffle=False),
test_dataloader=dict(samples_per_gpu=1, workers_per_gpu=5, shuffle=False),
train=[
dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_occ=True),
dict(
type='ColorJitter',
brightness=0.5,
contrast=0.5,
saturation=0.5,
hue=0.5),
dict(type='RandomGamma', gamma_range=(0.7, 1.5)),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='RandomFlip', prob=0.5, direction='horizontal'),
dict(type='RandomFlip', prob=0.5, direction='vertical'),
dict(
type='RandomAffine',
global_transform=dict(
translates=(0.05, 0.05),
zoom=(1.0, 1.5),
shear=(0.86, 1.16),
rotate=(-10.0, 10.0)),
relative_transform=dict(
translates=(0.00375, 0.00375),
zoom=(0.985, 1.015),
shear=(1.0, 1.0),
rotate=(-1.0, 1.0))),
dict(type='RandomCrop', crop_size=(384, 768)),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['imgs', 'flow_gt'],
meta_keys=[
'img_fields', 'ann_fields', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'filename_flow',
'ori_filename_flow', 'ori_shape', 'img_shape',
'img_norm_cfg'
])
],
data_root='data/Sintel',
test_mode=False,
pass_style='final')
],
val=dict(
type='ConcatDataset',
datasets=[
dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'ori_shape',
'img_shape', 'img_norm_cfg', 'scale_factor',
'pad_shape'
])
],
data_root='data/Sintel',
test_mode=True,
pass_style='clean'),
dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'ori_shape',
'img_shape', 'img_norm_cfg', 'scale_factor',
'pad_shape'
])
],
data_root='data/Sintel',
test_mode=True,
pass_style='final')
],
separate_eval=True),
test=dict(
type='ConcatDataset',
datasets=[
dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'ori_shape',
'img_shape', 'img_norm_cfg', 'scale_factor',
'pad_shape'
])
],
data_root='data/Sintel',
test_mode=True,
pass_style='clean'),
dict(
type='Sintel',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='InputResize', exponent=4),
dict(
type='Normalize',
mean=[0.0, 0.0, 0.0],
std=[255.0, 255.0, 255.0],
to_rgb=False),
dict(type='TestFormatBundle'),
dict(
type='Collect',
keys=['imgs'],
meta_keys=[
'flow_gt', 'filename1', 'filename2',
'ori_filename1', 'ori_filename2', 'ori_shape',
'img_shape', 'img_norm_cfg', 'scale_factor',
'pad_shape'
])
],
data_root='data/Sintel',
test_mode=True,
pass_style='final')
],
separate_eval=True))
optimizer = dict(
type='Adam', lr=3e-05, weight_decay=0.0004, betas=(0.9, 0.999))
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='MultiStage',
by_epoch=False,
gammas=[0.5, 0.5],
milestone_lrs=[3e-05, 2e-05],
milestone_iters=[0, 150000],
steps=[[
45000, 65000, 85000, 95000, 97500, 100000, 110000, 120000, 130000,
140000
],
[
195000, 215000, 235000, 245000, 247500, 250000, 260000, 270000,
280000, 290000
]])
runner = dict(type='IterBasedRunner', max_iters=300000)
checkpoint_config = dict(by_epoch=False, interval=50000)
evaluation = dict(interval=50000, metric='EPE')
log_config = dict(
interval=50,
hooks=[dict(type='TextLoggerHook'),
dict(type='TensorboardLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = 'https://download.openmmlab.com/mmflow/pwcnet/pwcnet_ft_4x1_300k_sintel_384x768.pth'
resume_from = None
workflow = [('train', 1)]
model = dict(
type='PWCNet',
encoder=dict(
type='PWCNetEncoder',
in_channels=3,
net_type='Basic',
pyramid_levels=[
'level1', 'level2', 'level3', 'level4', 'level5', 'level6'
],
out_channels=(16, 32, 64, 96, 128, 196),
strides=(2, 2, 2, 2, 2, 2),
dilations=(1, 1, 1, 1, 1, 1),
act_cfg=dict(type='LeakyReLU', negative_slope=0.1)),
decoder=dict(
type='PWCNetDecoder',
in_channels=dict(
level6=81, level5=213, level4=181, level3=149, level2=117),
flow_div=20.0,
corr_cfg=dict(type='Correlation', max_displacement=4, padding=0),
warp_cfg=dict(type='Warp', align_corners=True, use_mask=True),
act_cfg=dict(type='LeakyReLU', negative_slope=0.1),
scaled=False,
post_processor=dict(type='ContextNet', in_channels=565),
flow_loss=dict(
type='MultiLevelEPE',
p=1,
q=0.4,
eps=0.01,
reduction='sum',
weights=dict(
level2=0.005,
level3=0.01,
level4=0.02,
level5=0.08,
level6=0.32))),
train_cfg=dict(),
test_cfg=dict(),
init_cfg=dict(
type='Kaiming',
nonlinearity='leaky_relu',
layer=['Conv2d', 'ConvTranspose2d'],
mode='fan_in',
bias=0))
sys.platform: linux
Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.2.r11.2/compiler.29618528_0
GPU 0: Quadro RTX 4000
GCC: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
PyTorch: 1.8.0
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) oneAPI Math Kernel Library Version 2021.3-Product Build 20210617 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 10.2
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
- CuDNN 7.6.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.9.0
OpenCV: 4.5.3
MMCV: 1.3.15
MMFlow: 0.1.0+607e1ed
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.2
After visualizing optical flow as RGB video by visualize_flow(), is it possible to get optical flow from the converted RGB video?
from mmflow.apis import inference_model, init_model config_file = 'configs/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.py' checkpoint_file = 'checkpoints/pwcnet_ft_4x1_300k_sintel_final_384x768.pth' device = 'cuda:0' model = init_model(config_file, checkpoint_file, device=device) inference_model(model, 'demo/frame_0001.png', 'demo/frame_0002.png')
python demo/video_demo.py "/home/thhicv/program/mmflow/demo/demo.mp4" 'configs/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.py' 'checkpoints/pwcnet_ft_4x1_300k_sintel_final_384x768.pth' /home/thhicv/program/mmflow/demo/xx1x.mp4
/home/thhicv/anaconda3/envs/mmflow/bin/python /home/thhicv/program/mmflow/demo/video_demo.py /home/thhicv/视频/flow/溪丁灯.mp4 configs/pwcnet/pwcnet_ft_4x1_300k_sintel_final_384x768.py checkpoints/pwcnet_ft_4x1_300k_sintel_final_384x768.pth /home/thhicv/视频/flow/溪丁灯.mp4_flow.mp4
Traceback (most recent call last): File "/home/thhicv/program/mmflow/demo/video_demo.py", line 143, in <module> main(args) File "/home/thhicv/program/mmflow/demo/video_demo.py", line 83, in main result = inference_model(model, img1, img2) File "/home/thhicv/program/mmflow/mmflow/apis/inference.py", line 115, in inference_model data = test_pipeline(data) File "/home/thhicv/program/mmflow/mmflow/datasets/pipelines/compose.py", line 42, in __call__ data = t(data) File "/home/thhicv/program/mmflow/mmflow/datasets/pipelines/transforms.py", line 428, in __call__ imgs, scale_factor = self._resize_img(imgs) File "/home/thhicv/program/mmflow/mmflow/datasets/pipelines/transforms.py", line 451, in _resize_img img_ = mmcv.imresize(img, (newW, newH), return_scale=False) File "/home/thhicv/anaconda3/envs/mmflow/lib/python3.7/site-packages/mmcv/image/geometric.py", line 89, in imresize img, size, dst=out, interpolation=cv2_interp_codes[interpolation]) cv2.error: OpenCV(4.5.3) /tmp/pip-req-build-l1r0y34w/opencv/modules/imgproc/src/resize.cpp:3688: error: (-215:Assertion failed) !dsize.empty() in function 'resize'
Good job, grab the couch!!!
Good job, but the coder needs to practice programming skills. You could refer to Ao Yile ([email protected])
for your programming improvement. AYL yyds
where file is about this part?
thank you very much.
python tools/train.py configs/raft/raft_8x2_100k_flyingchairs_368x496.py --gpu-ids 0
The error is RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
python tools/train.py configs/raft/raft_8x2_100k_flyingchairs_368x496.py --gpu-ids 0 --launcher pytorch
The error becomes:
rank = int(os.environ['RANK'])
File "/home/miniconda3/envs/mmflow/lib/python3.8/os.py", line 675, in __getitem__
raise KeyError(key) from None
KeyError: 'RANK'
if 'RANK' not in os.environ:
os.environ['RANK'] = str(args.local_rank)
The error is:
ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable WORLD_SIZE expected, but not set
if 'WORLD_SIZE' not in os.environ:
os.environ['WORLD_SIZE'] = str(1)
The error is:
ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set
I got this error:
Traceback (most recent call last):
File "tools/train.py", line 199, in <module>
main()
File "tools/train.py", line 188, in main
train_model(
File "/home/framework/optical_flow/mmflow/mmflow/apis/train.py", line 212, in train_model
runner.run(data_loaders, cfg.workflow)
File "/home/miniconda3/envs/mmflow/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/home/miniconda3/envs/mmflow/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 67, in train
self.call_hook('after_train_iter')
File "/home/miniconda3/envs/mmflow/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "/home/miniconda3/envs/mmflow/lib/python3.8/site-packages/mmcv/runner/hooks/optimizer.py", line 56, in after_train_iter
runner.outputs['loss'].backward()
File "/home/miniconda3/envs/mmflow/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/miniconda3/envs/mmflow/lib/python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2, 128, 46, 62]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Hope I have addressed the issue clear enough. Thank for your help
i need that:
img = torch.rand(1, 6, 640,640).cuda()
output1 = model(img)
but the output is a dict,where can i modify this to a tensor?
parser.add_argument('--config',default='configs/liteflownet2/liteflownet2_ft_4x1_500k_kitti_320x896.py', help='Config file')
parser.add_argument('--checkpoint', default='models/liteflownet2/liteflownet2_ft_4x1_500k_kitti_320x896.pth', help='Checkpoint file')
Traceback (most recent call last):
File "/home/wangzhaoming/workspace/code/mmflow/demo/image_demo.py", line 44, in
main(args)
File "/home/wangzhaoming/workspace/code/mmflow/demo/image_demo.py", line 35, in main
result = inference_model(model, 'demo/frame_0001.png', 'demo/frame_0002.png')
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/apis/inference.py", line 115, in inference_model
data = test_pipeline(data)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/datasets/pipelines/compose.py", line 42, in call
data = t(data)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/datasets/pipelines/loading.py", line 146, in call
results = self._load_sparse_flow(results)
File "/disk/workspace/wangzhaoming/code/mmflow/mmflow/datasets/pipelines/loading.py", line 190, in _load_sparse_flow
filenames = list(results['ann_info'].keys())
KeyError: 'ann_info'
PyTorch: 1.11
CudaToolKits: 11.3.1
Error occur while running this command
python tools/train.py configs/raft/raft_8x2_50k_kitti2015_288x960.py
Complete stacktrace
/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Error detected in ReluBackward0. Traceback of forward call that caused the error:
File "tools/train.py", line 209, in <module>
main()
File "tools/train.py", line 205, in main
meta=meta)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/apis/train.py", line 238, in train_model
runner.run(data_loaders, cfg.workflow)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 61, in train
outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 75, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/flow_estimators/base.py", line 90, in train_step
losses = self(**data, test_mode=False)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/flow_estimators/base.py", line 59, in forward
return self.forward_train(*args, **kwargs)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/flow_estimators/raft.py", line 107, in forward_train
feat1, feat2, h_feat, cxt_feat = self.extract_feat(imgs)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/flow_estimators/raft.py", line 74, in extract_feat
cxt_feat = self.context(img1)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/encoders/raft_encoder.py", line 296, in forward
x = res_layer(x)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/container.py", line 141, in forward
input = module(input)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/utils/res_layer.py", line 88, in forward
out = _inner_forward(x)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/models/utils/res_layer.py", line 76, in _inner_forward
out = self.relu(out)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 98, in forward
return F.relu(input, inplace=self.inplace)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/nn/functional.py", line 1442, in relu
result = torch.relu(input)
(Triggered internally at /opt/conda/conda-bld/pytorch_1646755953518/work/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
File "tools/train.py", line 209, in <module>
main()
File "tools/train.py", line 205, in main
meta=meta)
File "/media/exthdd/laizeqiang/lzq/projects/misc/mmflow/mmflow/apis/train.py", line 238, in train_model
runner.run(data_loaders, cfg.workflow)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 67, in train
self.call_hook('after_train_iter')
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/mmcv/runner/hooks/optimizer.py", line 56, in after_train_iter
runner.outputs['loss'].backward()
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/laizeqiang/miniconda3/envs/openmmlab/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward
allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2, 128, 36, 120]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Hi
I wonder if it's possible to fine tune a pre-trained model on a dataset w/o flow labels, i.e., mixing supervised optical flow with unsupervised learning?
Thanks,
Alex
Reopening - links broken again?
Originally posted by @jhlusko in #12 (comment)
Would you provide support for scene flow algorithms like FlowNet3D, RAFT-3D, HPLFlowNet, PointPWC, etc?
Describe the bug
Good evening, i was wondering if some high level functions are provided to perform feature extraction on a model as RAFT.
I'd like to load the pretrained model on Things3d+Chairs and update only the final layer of it.
This issue will participate in the chinese GLCC activity.
基于MMFlow,实现 flow1d 算法(ICCV 2021 oral)
MMFlow是一款基于 PyTorch 和 MMCV 的光流估计开源工具箱,提供了多个 SOTA 光流估计算法,并支持光流领域主流学术数据集,以及光流可视化和评估方法。
通过在 MMFlow 中实现 flow1d 算法模型,你将可以了解光流领域的前沿知识,熟练掌握 MM 系列框架的整体流程,并积累深度学习相关的学术及工程经验。
在 MMFlow 中实现flow1d的算法模型,并复现其训练精度。
GitHub id: @MeowZheng
Email: [email protected]
对活动感兴趣的同学欢迎 OpenMMLab x GitLink 编程夏令营群聊
spynet is a tiny optical flow which compute efficient, and it is also used by like model like basicVSR.
Would it be added to mmflow?
Hi! Thanks for releasing such a wonderful toolbox. Given that MMSeg and MMFlow are in place, I wonder if there are any plans on making MMDepth that tackles depth estimation? I'd be very interested in using it.
window10
cuda10.2
pytorch1.6
python3.7
mmcv1.1.5
my first try failed. is the version of mmcv too old?
error info:
ImportError: cannot import name 'Correlation' from 'mmcv.ops'
Thanks for your error report and we appreciate it a lot.
Checklist
Describe the bug
A clear and concise description of what the bug is.
Reproduction
A placeholder for the command.
Environment
python mmdet/utils/collect_env.py
to collect necessary environment information and paste it here.$PATH
, $LD_LIBRARY_PATH
, $PYTHONPATH
, etc.)Error traceback
If applicable, paste the error trackback here.
A placeholder for trackback.
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.