colin97 / msn-point-cloud-completion

Morphing and Sampling Network for Dense Point Cloud Completion (AAAI2020)

MSN: Morphing and Sampling Network for Dense Point Cloud Completion

[paper] [data]

MSN is a learning-based shape completion method which can preserve the known structures and generate dense and evenly distributed point clouds. See our AAAI 2020 paper for more details.

In this project, we also provide an implementation for the Earth Mover's Distance (EMD) of point clouds, which is based on the auction algorithm and only needs $O(n)$ memory.

with 32,768 points after completion

Usage

1) Envrionment & prerequisites

• Pytorch 1.2.0
• CUDA 10.0
• Python 3.7
• Visdom
• Open3D

2) Compile

Compile our extension modules:

cd emd
python3 setup.py install
cd expansion_penalty
python3 setup.py install
cd MDS
python3 setup.py install

3) Download data and trained models

Download the data and trained models from here. We don't provide the partial point clouds of the training set due to the large size. If you want to train the model, you can generate them with the code and ShapeNetCore.v1. We generate 50 partial point clouds for each CAD model.

4) Train or validate

Run python3 val.py to validate the model or python3 train.py to train the model from scratch.

EMD

We provide an EMD implementation for point cloud comparison, which only needs $O(n)$ memory and thus enables dense point clouds (with 10,000 points or over) and large batch size. It is based on an approximated algorithm (auction algorithm) and cannot guarantee a (but near) bijection assignment. It employs a parameter $\epsilon$ to balance the error rate and the speed of convergence. Smaller $\epsilon$ achieves more accurate results, but needs a longer time for convergence. The time complexity is $O(n^2k)$, where $k$ is the number of iterations. We set a $\epsilon = 0.005, k = 50$ during training and a $\epsilon = 0.002, k = 10000$ during testing. Please refer toemd/README.md for more details.

Citation

If you find our work useful for your research, please cite:

@article{liu2019morphing,
  title={Morphing and Sampling Network for Dense Point Cloud Completion},
  author={Liu, Minghua and Sheng, Lu and Yang, Sheng and Shao, Jing and Hu, Shi-Min},
  journal={arXiv preprint arXiv:1912.00280},
  year={2019}
}

License

This project Code is released under the Apache License 2.0 (refer to the LICENSE file for details).

msn-point-cloud-completion's Issues

Run VAL on a cpu computer

Is it able to run VAL on a CPU computer? Look like I need an expansionPenaltyFunction class that supports CPU tensor

Is there a model trained with CD loss?

Hi, on your paper, you tested your model with Chamfer Distance as a loss function. Can you share the resulting model?

Also, Is there a reason that you created a new dataset rather than using the existing "PCN dataset"?

asking about generating training data

Hi,

After following the render process in https://github.com/wentaoyuan/pcn/tree/master/render, I can get those .pcd files, which should be the training data? Are there any other steps?

The completed data are provided in the google drive, so there is no need to follow the sample process in https://github.com/wentaoyuan/pcn/tree/master/sample?

Thank you!!

CUDA 10.1 support for EMD?

I successfully compiled EMD using CUDA 10.1 but I encountered some undefined symbol error. Does the code not support CUDA 10.1?

fail to run EMD "python setup.py install"

centos7
pytorch1.2
CUDA10.0
python 3.7

ERROR:
Traceback (most recent call last):
File "setup.py", line 2, in
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
ImportError: No module named torch.utils.cpp_extension

But I can run "import torch.utils.cpp_extension" by python in this environment

Visulization

Hi Colin,

Nice work!

Figure 6 is so beautiful, how do you draw it? Using matplotlib, cloudcompare or meshlab?

3D modeling file .obj normalization method

Hi,
I have a question about .obj file normalization method.
I want to test your deep-learning model on my 3D modeling dataset.
But, I don't find method that normalize .obj file size x, y, z components like your dataset(ShapeNet).
If you know about it, tell me anything.
Thank you.

Time for computing EMD

Thanks for your nice implementation.

I was very confused by the running time for computing EMD. When eps = 0.05, k = 1000, I got the following results. The size of input is (32, 16834, 3).

1. for the test_emd(), two sets of point cloud are generated randomly. Runtime is 3.57s and EMD is 0.037.
2. however, when computing the distance between real lidar point cloud and biased real lidar point cloud, runtime is 47s and EMD is 0.182. The input pc has been normalized to [0, 1].

The runtime is much longer with real data and could you please explain the difference?

Cuda running out of memory

I'm running val.py on a windows machine with a Nvidia 1060 6GB VRAM and I get the following:

RuntimeError: CUDA out of memory. Tried to allocate 1.56 GiB (GPU 0; 6.00 GiB total capacity; 2.12 GiB already allocated; 1.06 GiB free; 1.54 GiB cached)

How much VRAM did you have on your system?

Number of points in EMD function

Hi @Colin97,

Thanks for the great work and releasing the code.

I am using the EMD function and I found that it requires the number of input points to be divisible by 1024

class emdFunction(Function):
    @staticmethod
    def forward(ctx, xyz1, xyz2, eps, iters):

        batchsize, n, _ = xyz1.size()
        _, m, _ = xyz2.size()

        assert(n == m)
        assert(xyz1.size()[0] == xyz2.size()[0])
        assert(n % 1024 == 0)
        assert(batchsize <= 512)

MSN-Point-Cloud-Completion/emd/emd_module.py

Line 40 in c5f23a9

assert(n % 1024 == 0)

I am trying to compute the EMD distance between two point clouds contain 20 points each so I wonder can I remove this assert safely? or is there an easy way to modify the code for a 20-point point cloud?

Best,
Zirui

The install bug in 3090 GPU with cuda 11

_get_cuda_arch_flags
    raise ValueError("Unknown CUDA arch ({}) or GPU not supported".format(arch))

visualize

How to visualize the results

How to evaluate MSN on Completion 3d Benchmark？

Hi,
Thanks for your great work.

When I tried to evaluate MSN on the completion 3d benchmark, I found some difficulties, such as the version of cuda and pytorch. Completion 3d benchmark uses cuda9.0 and pytorch0.4.0.
Have you evaluated MSN on Completion 3d Benchmark?

EMD Normalisation

Hi Colin,

Awesome work !

I had a question regarding the normalisation of the point cloud for the EMD. Could you tell me why is it needed to normalise the coordinates between 0 and 1. What will happen if I don't do that ?

Transfer Learning

Hello @Colin97 , I was wondering if there's a way to perform transfer learning on the model you provided? Also, could you elaborate on the data prep part? I'm using ply files of my own, not from ShapeNet.

How to modify the code to make EMD work with points of other dimensions?

Great work and thanks for sharing the code!

I am wondering how to modify your code to work with points in a 4 dim space, i.e. points with shape [#batch, #points, 4].

It seems that I only need to modify some lines in <emd_cuda.cu>, but since I do not fimilar with cuda programming at all,

it will be appreciated if you can give me some detail advises!

eps and k values

Hi Colin,

I have an other question regarding the eps and k parameters for the EMD loss. Could you provide a more detail explanation of their purpose ?

Thanks,

Problem about installation

Thanks for your great work,
G++ error occurred when I entered python setup.py install.
running install running bdist_egg running egg_info writing emd.egg-info/PKG-INFO writing dependency_links to emd.egg-info/dependency_links.txt writing top-level names to emd.egg-info/top_level.txt listing git files failed - pretending there aren't any reading manifest file 'emd.egg-info/SOURCES.txt' writing manifest file 'emd.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_ext building 'emd' extension creating /home/lq/New_p/VRCNet-main/utils/emd/build creating /home/lq/New_p/VRCNet-main/utils/emd/build/temp.linux-x86_64-3.7 Emitting ninja build file /home/lq/New_p/VRCNet-main/utils/emd/build/temp.linux-x86_64-3.7/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) 1.10.2.git.kitware.jobserver-1 creating build/lib.linux-x86_64-3.7 g++ -pthread -shared -B /home/lq/anaconda3/envs/mvp/compiler_compat -L/home/lq/anaconda3/envs/mvp/lib -Wl,-rpath=/home/lq/anaconda3/envs/mvp/lib -Wl,--no-as-needed -Wl,--sysroot=/ /home/lq/New_p/VRCNet-main/utils/emd/build/temp.linux-x86_64-3.7/emd.o /home/lq/New_p/VRCNet-main/utils/emd/build/temp.linux-x86_64-3.7/emd_cuda.o -L/home/lq/anaconda3/envs/mvp/lib/python3.7/site-packages/torch/lib -L/usr/local/cuda-10.1/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.7/emd.cpython-37m-x86_64-linux-gnu.so g++: error: /home/lq/New_p/VRCNet-main/utils/emd/build/temp.linux-x86_64-3.7/emd.o: No such file or directory g++: error: /home/lq/New_p/VRCNet-main/utils/emd/build/temp.linux-x86_64-3.7/emd_cuda.o: No such file or directory error: command 'g++' failed with exit status 1

import error---undefined symbol

Does anyone meet this bug and know how to fix it?

Train on Custom Data

Hi has anyone been able to train MSN-PCC on their own custom dataset? If so, which categories and how well did it perform? Were there any dataset generation/loading problems?

How to change the EMD wrapper to deal with flexible number of points instead of 1024x?

Hi, Thank you for your good job! How to change the EMD wrapper to deal with flexible number of points, instead of just 1024x?

Question about MDS code

Hello~ Thank you for sharing your nice work!
I have two questions about the MDS_cuda.cu file.

1. In line 114, (float t = 5.0 * mean_mst_length[batch_index] * mean_mst_length[batch_index]; )
   I read your paper(arXiv ver.), and It seems that you used the mean_mst_length as parameter sigma(in the paper). But I can't understand why the coefficient is 5.0, not 2.0. Could you explain me about this?

2. In line 130, (temp[k] += k < 8192 ? exp(-d / t) : exp(-d / t) * 2.0;)
   Could you explain me why this line is needed?

About the dataset and the gpu.

It seems like that the partial point cloud dataset generated from ShapeNet is very large (up to 49 GB), so could you please give the information of the GPU you used in the experiment ShapeNet completion? Can we train the model with a general GPU?

Failed to establish a new connection: [Errno 111] Connection refused

Hi,
Thanks for your great work.
When I train the model, here are some logs about connection refused, just as the following Traceback shown.
But the model in training ... , do I need care the connection logs?

Best and Thanks

Traceback (most recent call last):
File "/home/yjcai/anaconda3/envs/msn/lib/python3.7/site-packages/visdom/init.py", line 711, in _send
data=json.dumps(msg),
File "/home/yjcai/anaconda3/envs/msn/lib/python3.7/site-packages/visdom/init.py", line 677, in _handle_post
r = self.session.post(url, data=data)
File "/home/yjcai/anaconda3/envs/msn/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/yjcai/anaconda3/envs/msn/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/yjcai/anaconda3/envs/msn/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/yjcai/anaconda3/envs/msn/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
object at 0x7f1a8f4b1ad0>: Failed to establish a new connection: [Errno 111] Connection refused'))
MSN_TRAIN train [0: 0/9] emd1: 0.341410 emd2: 0.302342 expansion_penalty: 0.003462
MSN_TRAIN train [0: 1/9] emd1: 0.321795 emd2: 0.290033 expansion_penalty: 0.003422
MSN_TRAIN train [0: 2/9] emd1: 0.347615 emd2: 0.314816 expansion_penalty: 0.003539
MSN_TRAIN train [0: 3/9] emd1: 0.331207 emd2: 0.294323 expansion_penalty: 0.003690
MSN_TRAIN train [0: 4/9] emd1: 0.306453 emd2: 0.276825 expansion_penalty: 0.003754
MSN_TRAIN train [0: 5/9] emd1: 0.277850 emd2: 0.246808 expansion_penalty: 0.003884
MSN_TRAIN train [0: 6/9] emd1: 0.303935 emd2: 0.266290 expansion_penalty: 0.003877
MSN_TRAIN train [0: 7/9] emd1: 0.293241 emd2: 0.256959 expansion_penalty: 0.004172
MSN_TRAIN train [0: 8/9] emd1: 0.285149 emd2: 0.239006 expansion_penalty: 0.004101
MSN_TRAIN train [0: 9/9] emd1: 0.309674 emd2: 0.286977 expansion_penalty: 0.004666

Save the output

Do you have a function to save the point cloud output?

Can I ask how do you draw those beautiful point cloud pictures in teaser.png?

Nothing's more refreshing than enjoying such a 3d models picture!

You also put some super nice point clouds in:
https://github.com/Colin97/MSN-Point-Cloud-Completion/blob/master/emd/CDEMD.png

I have seen similar point cloud images in some other papers, and I always wanted to know how to draw them. I would be very appreciate it if you could show it to me.

ヾ(≧▽≦*)o

Could you please give the python version and open3d version you used to generated the partial point cloud?

I try the code to generate the partial point clouds but got the failure. Could you please give more details about the environment for the code to generate the partial point clouds?

Do you visualise the results by Open3d ?

Denoise the generated partial point cloud for training.

Dear,

Thanks for your outstanding work!

I have followed the render process in PCN, and obtained the partial point cloud from ShapeNet for training. However, I find that some of my results are a little noisy as follows:

And I notice the val data you provide is title as 'model_id/scan_id_denoised.pcd', so I am wondering whether there is a further denosing step to be taken to refine the generated partial data?

Have a good day!

Can the dimension of data larger than 3 to calculate emd?

Hello, thanks for your wonderful work!

If I have two sets of data whose dimensions are larger than 3. Can I still use your emd to calculate their distance?

x1 = torch.rand(20, 8192, 100).cuda()
x2 = torch.rand(20, 8192, 100).cuda()

emd = emdModule()
dis, assigment = emd(x1, x2, 0.05, 3000)

I have tested it and no error is reported. But I am not sure if it makes sense. Thanks!

It seems that EMD and Verified EMD are different:

Input_size:  torch.Size([20, 8192, 100])
Runtime: 0.863681s
EMD: 0.046708
|set(assignment)|: 8192
Verified EMD: 4.075622

The PCN model you compared with is trained by yourself?

Is The PCN model you compared with trained by yourself? Or you just use the pretrained model released by the PCN official? And I want to know the PCN you used is PCN-CD or PCN-EMD.

comparing point clouds

Thanks for your repository.
Do you think it is possible to use your code to calcule the distances between two point clouds ?
I have S_A and S_B are two point clouds sampled on the same surface (at two different time, the surface may be deformed or translated/rotated).
And I want to calculate the distance between each point in S_B with the corresponding implicit surface of S_A (an example is a monitoring project to detect changes between two point clouds)
Thanks very much

How do I run this with my own data and just infer?

Hey all,

I see that you have a custom dataloader for the solution, but I was curious how to use this network on my own data. For example, say I have a PCD which is already incomplete, I do not have the ground truth for this. I just want to run inference on this data. How would be the best way to progress using this network? Any help on this would be very much appreciated. Thank you for the contribution !!

Failed to run EMD

Hi,

I tried to run emd_module/test_emd(). However, after "python3 setup.py install" and run emd_module.py, it shows "AttributeError: module 'emd' has no attribute 'forward'", at line 58.

Any possible reason about it?

Here's the configuration of my machin:
Python 3.7
Pytorch 1.3.0
CUDA 9.2

Thanks!

Generalization capability

Just out of curiosity, have you tried training jointly on multiple categories and/or tested on unknown categories?

Of course results would degrade, but would it still give meaningful results?

Thanks!

AttributeError: module 'MDS' has no attribute 'minimum_density_sampling'

When I try to train the network, I got the following error.

Namespace(batchSize=16, env='MSN_TRAIN', model='', n_primitives=16, nepoch=25, num_points=8192, workers=24)
Exception in user code:
------------------------------------------------------------
Random Seed:  1957
Train Set Size:  28974
Traceback (most recent call last):
  File "train.py", line 121, in <module>
    output1, output2, emd1, emd2, expansion_penalty  = network(input, gt.contiguous(), 0.005, 50)         
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "train.py", line 38, in forward
    output1, output2, expansion_penalty = self.model(inputs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/xiehaozhe/Development/MSN-Point-Cloud-Completion/model.py", line 168, in forward
    resampled_idx = MDS_module.minimum_density_sample(xx[:, 0:3, :].transpose(1, 2).contiguous(), out1.shape[1], mean_mst_dis) 
  File "./MDS/MDS_module.py", line 29, in forward
    MDS.minimum_density_sampling(xyz, npoint, mean_mst_length, idx)
AttributeError: module 'MDS' has no attribute 'minimum_density_sampling'

Assert error: assert(primitive_size <= 512)

I got the following errors when setting n_pts to 16384.

Namespace(batchSize=16, env='MSN_TRAIN', model='', n_primitives=16, nepoch=25, num_points=16384, workers=24)
Random Seed:  3764
Train Set Size:  28974
Traceback (most recent call last):
  File "train.py", line 121, in <module>
    output1, output2, emd1, emd2, expansion_penalty  = network(input, gt.contiguous(), 0.005, 50)         
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "train.py", line 38, in forward
    output1, output2, expansion_penalty = self.model(inputs)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/lustre/xiehaozhe/Development/MSN-Point-Cloud-Completion/model.py", line 159, in forward
    dist, _, mean_mst_dis = self.expansion(out1, self.num_points//self.n_primitives, 1.5)
  File "/mnt/lustre/xiehaozhe/Applications/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "./expansion_penalty/expansion_penalty_module.py", line 53, in forward
    return expansionPenaltyFunction.apply(input, primitive_size, alpha)
  File "./expansion_penalty/expansion_penalty_module.py", line 27, in forward
    assert(primitive_size <= 512)
AssertionError

Training Time

Hi, thanks for your impressive work, how about your training time over the whole dataset you provided (about 30000 shapes and 50 views per shape)? @Colin97

Backward call fails

Hello,

i have an issue when trying to train the network. The loss_net.backward() call fails with this pytorch error

RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false.

But so far I was not able to find the reason for this error. Has someone else experienced this issue yet?

Thank you very much

Data for the test set of ShapeNet

Can you provide the preprocessed data for the test set of ShapeNet which contains 1200 models?

Image in the paper

Hello, how do you display the point cloud image in the paper? Can you publish your code?