yangyanli / pointcnn

PointCNN: Convolution On X-Transformed Points (NeurIPS 2018)

PointCNN: Convolution On X-Transformed Points

Created by Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.

Introduction

PointCNN is a simple and general framework for feature learning from point cloud, which refreshed five benchmark records in point cloud processing (as of Jan. 23, 2018), including:

• classification accuracy on ModelNet40 (91.7%, with 1024 input points only)
• classification accuracy on ScanNet (77.9%)
• segmentation part averaged IoU on ShapeNet Parts (86.13%)
• segmentation mean IoU on S3DIS (65.39%)
• per voxel labelling accuracy on ScanNet (85.1%)

See our preprint on arXiv (accepted to NeurIPS 2018) for more details.

Pretrained models can be downloaded from here.

Performance on Recent Benchmarks

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

ABC: A Big CAD Model Dataset For Geometric Deep Learning

Practical Applications

3D cities: Deep Learning in three-dimensional space (from Esri)

PointCNN: replacing 50,000 man hours with AI (from Esri)

Point Cloud Segmentation using PointCNN in ArcGIS API for Python (from Esri)

More Implementations

• Pytorch implementation from PyTorch Geometric
• Pytorch implementation from Berkeley CS294-131 Course Proj
• MXNet implementation
• Jittor implementation

We highly welcome issues, rather than emails, for PointCNN related questions.

License

Our code is released under MIT License (see LICENSE file for details).

Code Organization

The core X-Conv and PointCNN architecture are defined in pointcnn.py.

The network/training/data augmentation hyper parameters for classification tasks are defined in pointcnn_cls, for segmentation tasks are defined in pointcnn_seg.

Explanation of X-Conv and X-DeConv Parameters

Take the xconv_params and xdconv_params from shapenet_x8_2048_fps.py for example:

xconv_param_name = ('K', 'D', 'P', 'C', 'links')
xconv_params = [dict(zip(xconv_param_name, xconv_param)) for xconv_param in
                [(8, 1, -1, 32 * x, []),
                 (12, 2, 768, 32 * x, []),
                 (16, 2, 384, 64 * x, []),
                 (16, 6, 128, 128 * x, [])]]

xdconv_param_name = ('K', 'D', 'pts_layer_idx', 'qrs_layer_idx')
xdconv_params = [dict(zip(xdconv_param_name, xdconv_param)) for xdconv_param in
                 [(16, 6, 3, 2),
                  (12, 6, 2, 1),
                  (8, 6, 1, 0),
                  (8, 4, 0, 0)]]

Each element in xconv_params is a tuple of (K, D, P, C, links), where K is the neighborhood size, D is the dilation rate, P is the representative point number in the output (-1 means all input points are output representative points), and C is the output channel number. The links are used for adding DenseNet style links, e.g., [-1, -2] will tell the current layer to receive inputs from the previous two layers. Each element specifies the parameters of one X-Conv layer, and they are stacked to create a deep network.

Each element in xdconv_params is a tuple of (K, D, pts_layer_idx, qrs_layer_idx), where K and D have the same meaning as that in xconv_params, pts_layer_idx specifies the output of which X-Conv layer (from the xconv_params) will be the input of this X-DeConv layer, and qrs_layer_idx specifies the output of which X-Conv layer (from the xconv_params) will be forwarded and fused with the output of this X-DeConv layer. The P and C parameters of this X-DeConv layer is also determined by qrs_layer_idx. Similarly, each element specifies the parameters of one X-DeConv layer, and they are stacked to create a deep network.

PointCNN Usage

PointCNN is implemented and tested with Tensorflow 1.6 in python3 scripts. Tensorflow before 1.5 version is not recommended, because of API. It has dependencies on some python packages such as transforms3d, h5py, plyfile, and maybe more if it complains. Install these packages before the use of PointCNN.

If you can only use Tensorflow 1.5 because of OS factor(UBUNTU 14.04),please modify "isnan()" to "std::nan()" in "/usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/numeric_types.h" line 49

Here we list the commands for training/evaluating PointCNN on classification and segmentation tasks on multiple datasets.

• Classification

  • ModelNet40

  cd data_conversions
  python3 ./download_datasets.py -d modelnet
  cd ../pointcnn_cls
  ./train_val_modelnet.sh -g 0 -x modelnet_x3_l4
  

  • ScanNet

  Please refer to http://www.scan-net.org/ for downloading ScanNet task data and scannet_labelmap, and refer to https://github.com/ScanNet/ScanNet/tree/master/Tasks/Benchmark for downloading ScanNet benchmark files:

  scannet_dataset_download

  |_ data

  |_ scannet_labelmap

  |_ benchmark

  cd ../data/scannet/scannet_dataset_download/
  mv ./scannet_labelmap/scannet-labels.combined.tsv ../benchmark/
  
  #./pointcnn_root
  cd ../../../pointcnn/data_conversions
  python extract_scannet_objs.py -f ../../data/scannet/scannet_dataset_download/data/ -b ../../data/scannet/scannet_dataset_download/benchmark/ -o ../../data/scannet/cls/
  python prepare_scannet_cls_data.py -f ../../data/scannet/cls/
  cd ../pointcnn_cls/
  ./train_val_scannet.sh -g 0 -x scannet_x3_l4
  

  • tu_berlin

  cd data_conversions
  python3 ./download_datasets.py -d tu_berlin
  python3 ./prepare_tu_berlin_data.py -f ../../data/tu_berlin/ -a --create-train-test
  cd ../pointcnn_cls
  ./train_val_tu_berlin.sh -g 0 -x tu_berlin_x3_l4
  

  • quick_draw

  Note that the training/evaluation of quick_draw requires LARGE RAM, as we load all stokes into RAM and converting them into point cloud on-the-fly.

  cd data_conversions
  python3 ./download_datasets.py -d quick_draw
  cd ../pointcnn_cls
  ./train_val_quick_draw.sh -g 0 -x quick_draw_full_x2_l6
  

  • MNIST

  cd data_conversions
  python3 ./download_datasets.py -d mnist
  python3 ./prepare_mnist_data.py -f ../../data/mnist
  cd ../pointcnn_cls
  ./train_val_mnist.sh -g 0 -x mnist_x2_l4
  

  • CIFAR-10

  cd data_conversions
  python3 ./download_datasets.py -d cifar10
  python3 ./prepare_cifar10_data.py
  cd ../pointcnn_cls
  ./train_val_cifar10.sh -g 0 -x cifar10_x3_l4
  

• Segmentation

  We use farthest point sampling (the implementation from PointNet++) in segmentation tasks. Compile FPS before the training/evaluation:

  cd sampling
  bash tf_sampling_compile.sh
  

  • ShapeNet

  cd data_conversions
  python3 ./download_datasets.py -d shapenet_partseg
  python3 ./prepare_partseg_data.py -f ../../data/shapenet_partseg
  cd ../pointcnn_seg
  ./train_val_shapenet.sh -g 0 -x shapenet_x8_2048_fps
  ./test_shapenet.sh -g 0 -x shapenet_x8_2048_fps -l ../../models/seg/pointcnn_seg_shapenet_x8_2048_fps_xxxx/ckpts/iter-xxxxx -r 10
  cd ../evaluation
  python3 eval_shapenet_seg.py -g ../../data/shapenet_partseg/test_label -p ../../data/shapenet_partseg/test_data_pred_10 -a
  

  • S3DIS

  Please refer to data_conversions for downloading S3DIS, then:

  cd data_conversions
  python3 prepare_s3dis_label.py
  python3 prepare_s3dis_data.py
  python3 prepare_s3dis_filelists.py
  mv S3DIS_files/* ../../data/S3DIS/out_part_rgb/
  ./train_val_s3dis.sh -g 0 -x s3dis_x8_2048_fps -a 1
  ./test_s3dis.sh -g 0 -x s3dis_x8_2048_fps -a 1 -l ../../models/seg/s3dis_x8_2048_fps_xxxx/ckpts/iter-xxxxx -r 4
  cd ../evaluation
  python3 s3dis_merge.py -d <path to *_pred.h5>
  python3 eval_s3dis.py
  

We use a hidden marker file to note when prepare is finished to avoid re-processing. This cache can be invalidated by deleting the markers.

Please notice that these command just for Area 1 (specified by -a 1 option) validation. Results on other Areas can be computed by iterating -a option.

• ScanNet

Please refer to data_conversions for downloading ScanNet, then:

cd data_conversions
python3 prepare_scannet_seg_data.py
python3 prepare_scannet_seg_filelists.py
cd ../pointcnn_seg
./train_val_scannet.sh -g 0 -x scannet_x8_2048_k8_fps
./test_scannet.sh -g 0 -x scannet_x8_2048_k8_fps -l ../../models/seg/pointcnn_seg_scannet_x8_2048_k8_fps_xxxx/ckpts/iter-xxxxx -r 4
cd ../evaluation
python3 eval_scannet.py -d <path to *_pred.h5> -p <path to scannet_test.pickle>

• Semantic3D

Please check the free disk space before start, about 900 GB will be required.

cd data_conversions
bash download_semantic3d.sh
bash un7z_semantic3d.sh
python3 prepare_semantic3d_data.py
mkdir ../../data/semantic3d/filelists
python3 prepare_semantic3d_filelists.py
cd ../pointcnn_seg
./train_val_semantic3d.sh -g 0 -x semantic3d_x4_2048_fps
./test_semantic3d.sh -g 0 -x semantic3d_x4_2048_fps -l <path to ckpt>
cd ../evaluation
python3 semantic3d_merge.py -d <path to *_pred.h5> -v <reduced or full>

• Tensorboard

  If you want to monitor your train step, we recommend you use the following command

  cd <your path>/PointCNN
  tensorboard --logdir=../models/<seg/cls> <--port=6006>
  

pointcnn's Issues

The problem about shapeNet

When I run this command"./test_shapenet.sh -g 0 -x shapenet_x8_2048_fps -l ../../models/seg/pointcnn_seg_shapenet_x8_2048_fps_xxxx/ckpts/iter-xxxxx -r 10"
The error occured in test_seg.py line 94:probs_op = net.probs.It tells me that no probs attribute in net.
The net is defined in pointcnn.py and I didn't find the self.probs in object point cnn.
And I found that in the previous version of point cnn.
If this is a bug, can you fix it?
Thank you!

What's the intention for map_fn

Hi, authors,

What's the purpose for map_fn, which we could find many places referring to? e.g. line 65 , line 96, line 114 in train_val_cls.py

test on shapenet

I try to test shape net dataset on the trained model, however, there is an error that when the code run into the pointcnn.py, the terminal tells me that the net has no attributes about probs(simply there is >>no net.probs in the test_seg.py ).
Could you tell me how to change the code to make it run?
Thanks a lot!

Concerns about classification-modelnet40

Thanks for sharing your great work!

I am a beginner of deep learning models used on point clouds. I am very interested about your achievement. However, I have one question when conducting Classification operation on ModelNet40. You said performed "/train_val_modelnet.sh -g 0 -x modelnet_x3_l4", and it output "Train/Val with setting modelnet_x3_l4 on GPU 0!", then nothing happened. I would like to know how to achieve classification function, can you give the details steps after getting the above message? My operating environment is Ubuntu16.04 + python3.5
Thank you for your kind assistance.

Where is the {scannet_labelmap}?

I want to try to make a classification for my point cloud data.

But you write: download [scannet_labelmap] from ScanNet, actually I can't find this file or folder, how can I operate to finish my classification?

Thanks for your attention!

Undefined name ‘tnet’ again

Reopening #2 because of the update at charlesq34/pointnet2#21

question on the "links" usage

Hi, @yangyanli ,

According to these lines in pointcnn.py, "links" are used there. Could you elaborate a bit on its function? Does it mean the connections/kernel_weights/feature_maps between two layers?

THX!

therre is a problem when i run train_val_cls.py

parser = argparse.ArgumentParser()
parser.add_argument('--path', '-t', help='Path to data', default='/home/data/modelnet40_ply_hdf5_2048/train_files.txt') 
parser.add_argument('--path_val', '-v', help='Path to validation data',default='/home/data/modelnet40_ply_hdf5_2048/test_files.txt')
parser.add_argument('--load_ckpt', '-l', help='Path to a check point file for load')
parser.add_argument('--save_folder', '-s', help='Path to folder for saving check points and summary', default='/home/data/save/')
parser.add_argument('--model', '-m', help='Model to use', default='pointcnn_cls')
parser.add_argument('--setting', '-x', help='Setting to use', default='modelnet_x3_l4_p032')
args = parser.parse_args()

i changed your code as above,and i got an error blow

InvalidArgumentError (see above for traceback): clip_value_min and clip_value_max must be either of the same shape as input, or a scalar. input shape: [128,32,3]clip_value_min shape: [1]clip_value_max shape: [1]
[[Node: jitter_clipped = ClipByValue[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](mul, mul_1, mul_2)]]

thanks for your help!

About prepare_multiChannel_seg_data.py

There are still bugs in prepare_multiChannel_seg_data.py？

about s3dis_upsampling.py

I am confused about line 95~99 in s3dis_upsampling.py:
for k, vid in enumerate(uvidx):
voxel_label_dict[vid] = seg[k]

for vid in vidx:
ori_seg.append(voxel_label_dict[vid])

Could you please tell me the function of these lines ?

And I trained the model on Area2～6 of S3DIS, then tested on Area1.
If I run s3dis_upsampling.py before running eval_s3dis.py, the output of eval_s3dis.py is:

[741860, 1081219, 927535, 993140, 209411, 126037, 231132, 105981, 218610, 214055, 21996, 204034, 70819]
[339878, 330121, 640186, 811327, 88785, 99274, 197668, 58156, 122522, 143403, 10943, 116397, 2187169]
[65307, 102514, 164596, 287139, 11102, 36893, 44290, 15898, 16273, 24691, 1482, 35241, 4100]
Overall accuracy: 0.1573169259996786
IoU:
0.06425128710163307
0.07832515552105475
0.11730672605790646
0.18923990066748916
0.03867026130814298
0.19580401023256802
0.11518556084367117
0.10724573155512382
0.050092501669955276
0.07419906421009295
0.04711193057189179
0.12357025141133982
0.001819078853962575
0.09252472769267937

But if I don't run s3dis_upsampling.py, the output of eval_s3dis.py will be greatly improved:

[741860, 1081219, 927535, 993140, 209411, 126037, 231132, 105981, 218610, 214055, 21996, 204034, 70819]
[332462, 634309, 554291, 726372, 88278, 83955, 164272, 49907, 104344, 120826, 9476, 95388, 2181949]
[275670, 627944, 547055, 626761, 75240, 76869, 150398, 45148, 86119, 103163, 6392, 80861, 19842]
Overall accuracy: 0.5288675546738922
IoU:
0.3451691099502662
0.5773751728602112
0.5852288956332621
0.5735625041752421
0.33823483135460264
0.5774283932904156
0.6138543545872346
0.40769369694780566
0.36362446428948425
0.44520926298345403
0.25486443381180224
0.369969939742223
0.008886098330173056
0.4200847044581674

Have you ever met this problem?

Problem in data_utils.py

line 113-114
In load_cls()

if 'normal' in data:
    points.append(np.concatenate([data['data'][...], data['data'][...]], axis=-1).astype(np.float32))

It seems if there are normal vectors in the file, coordinates will be used again as features for the the first X-Conv layer.

But in the paper, it says:

"For input point cloud without any additional features, i.e., F is empty, the first X-Conv layer uses only Fδ ."

So does this step improve the performance?
If I concatenate coordinates with normal vectors, will it help?

Problem with ScanNet Classification

Hi,

I'm having some issues with the preprocessing of ScanNet dataset for classification.

When I try to run the command:
python scannet_extract_obj.py -f ../../data/scannet/scannet_dataset_download/data/ -b ../../data/scannet/scannet_dataset_download/benchmark/ -o ../../data/scannet/cls/

The script crashes with this error:

Traceback (most recent call last):
File "scannet_extract_obj.py", line 375, in
main()
File "scannet_extract_obj.py", line 336, in main
sceneid = scene.strip().split("scene")[1]
IndexError: list index out of range

The scene variable when the crash occurs is with the string:
scene: ../../data/scannet/scannet_dataset_download/data//h5_scannet

Has anyone had the same problem?
Thanks

What's the meaning of "with_fps"

Hi, authors,

What's the meaning of with_fps, which found in modelnet_x2_l4.py? It'll be much useful if more comments could be appended for each main parameters in modelnet_x2_l4.py.

P.S. I finally found the meaning of fps. That's "Furthest point sampling" from here , right?

THANKS

Core Dump when import tf_sampling

Hi Yangyan,
I have compiled the tf_sampling module, however, when I try to import it, "segmentation fault (core dumped)" happens. Is there anything I need to pay attention of while compiling the module? (TF version, cuda,gcc version?)
Thanks,
Daxuan

What is the result of PointCNN in semantic3d dataset?

希望了解PointCNN在semantic3D数据集上的结果？以及训练这个数据集需要的时间？谢谢！

Problem when running train_val_modelnet.sh

I follow the command:

cd data_conversions
python3 ./download_datasets.py -d modelnet
cd ../pointcnn_cls
./train_val_modelnet.sh -g 0 -x modelnet_x2_l4

However, it will generate many model files iteratively. And the error is:

File "/home/bandary/.conda/envs/my_env/lib/python3.6/shutil.py", line 359, in copytree
raise Error(errors)
shutil.Error: [('/home/bandary/Documents/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_6110_2018-04-13-10-58-25/PointCNN/models/cls/pointcnn_cls_modelnet_x2_l4_

the code about shapenet segmentation

When I run the segmentation task on shapenet dataset, there is an error about max() is empty.
The code is in the prepre_partseg_data.py 55th line(category_label_seg_max_list is empty).
I am confused about it.
Thank you!

Could you offer the pretrained model for semantic segmentation task?

As the title says, could you offer the pre-trained model for semantic segmentation (on S3DIS and
Scannet ) prediction. We want to use the model as a prior for fulfilling follow-by other tasks.
Thanks !

train_val_shapenet.sh shapenet_x8_2048_fps error

Hi ,thanks for your share!
when i try to train the shapenet_x8_2048_fps,i accept a error which indicates that there is no "label_weights" in shapenet_x8_2048_fps.py. The code of train_val_seg.py does need the "label_weights" say label_weights_list = setting.label_weights .In your paper, it says

ShapeNet Parts contains 16, 880 models (14, 006/2, 874 training/ testing split) from 16 shape categories, each annotated with 2 to 6 parts and there are 50 different parts in total.

so, should i add a vector of 16 dims which named label_weights ?
Awaiting for your reply .Thank you very much

subvolume supervision solve overfitting

sorry,i want to know why the subvolume supervision can solve overfitting,i think,it only realizes the prediction of a whole through a part.

use_bias can be set to False when dense or conv followed by batchnorm

For example,

PointCNN/pointfly.py

Line 283 in f194cab

conv2d = tf.layers.conv2d(input, output, kernel_size=kernel_size, strides=(1, 1), padding='VALID',

Tensorflow's API use 'use_bias=True' as default.

s3dis

1. How to use eval_s3dis.py?
   We split each room into several small blocks for testing, right?
   so, do we need to combine the prediction of each block to get the prediction of the whole room?
   since the groud-truth-files used in "eval_s3dis.py" is about the whole room.
   if so, can you provide the script for combining the predictions of each room blocks?

2. In s3dis_split.py.
   we first "split-to-blocks" and then "reblock",
   so is there any chance the number of points in the blocks still less than "block_min_pnum"
   after the "reblock" operation?
   of course, I think it does not matter, so you can choose to ignore this question.

3. when using "eval_shapenet_seg.py" in the shapenet part segmentation task,
   we have to change the following command :
   python3 eval_shapenet_seg.py -g ../../data/shapenet_partseg/test_label -p ../../data/shapenet_partseg/test_data_pred_10
   into:
   python3 eval_shapenet_seg.py -g ../../data/shapenet_partseg/test_label -p ../../data/shapenet_partseg/test_data_pred_10 -a
   right?

Thank you for your great work!

Per point segmentation for shapenet

Hey Guys,

I am playing around with PointCNN on my own dataset which is pretty similar to shapenet. I may wonder if it could be possible to get a per point segmentation for each pointcloud? I just tested it with my own test.h5 files but it seems that I can only get the category (class) for each pointcloud.
Correct me if I am wrong :)

Many Thanks so far!

Greetings, R.

Some questions about the result of semantic segmentation

Hi！
I tried to use the PointCNN to process ourdoor scene point clouds. I want to get the result of point clouds semantic segmentation in outdoor scene. I input my point clouds with labels, and after iterative train got the model. And I get this result:

I feel that this is a voxel processing of the point cloud data in the original scene, because the output results appear to be voxel. The accuracy of the result is not high.
I used the S3DIS repo to evaluate. I want to try semantic3D repo, but it seems like a file is missing(test_semantic3d.sh) Could you add this file?

./train_val_modelnet.sh: line 40: ../../models/cls/pointcnn_cls_modelnet_x2_l4.txt: No such file or directory

您好！出现以上问题，求解决方案

I wrote a a MXNet implemetation of your works.

Sincere thanks to your open code.
https://github.com/chinakook/PointCNN.MX

I found some bugs!!

Recently I studied the semantic segmentation of point cloud data. I mainly process point clouds in outdoor scenes. I found that this repo has been changed very frequently. When I run test_s3dis.sh in S3DIS, I found some errors:

1. In your README.md, 162 line, "./test_s3dis.sh -g 0 -x s3dis_x8_2048_fps_k16 -l ../../models/seg/s3dis_x8_2048_fps_k16_xxxx/ckpts/iter-xxxxx -r 4", I notice the "s3dis_x8_2048_fps_k16" is wrong, because in "pointcnn_seg" folder, there is "s3dis_x8_2048_k16_fps"
2. In test_s3dis.sh, 68 line, "CUDA_VISIBLE_DEVICES=$gpu python3 ../test_geneal_seg.py -t ../../data/S3DIS/out_part_rgb/val1.txt -f ../../data/S3DIS/out_part_rgb/Area1_data/ -c ../../data/S3DIS/out_part_rgb/categories.txt -l $ckpt -m pointcnn_seg -x $setting -r $repeat $save_ply"
   “test_geneal_seg.py” miss "r" in "general"
3. In "test_general_seg.py", 91 line, "net = model.Net(points_sampled, features_sampled, num_class, is_training, setting)" a parameter "num_class" is may be redundant， because in pointcnn_seg.py, 10 line, " def init(self, points, features, is_training, setting):" without "num_class" parameter.
   I also found some other errors in the process of running……
   I would like to ask a few questions:
4. In "#pointcnn-master#/data_conversions/split_data/s3dis_split.py" , Can you describe the idea of dividing point cloud data into space volexs for quantitative statistics? And why do you want to downsampling? Can the final test result restore the original point cloud number?
   I'm running point cloud data on servers in our lab and I hope to keep in touch with your team.
   My e-mail: [email protected]

Feed different group of points into independent PointCNNs parallelly

Thanks for your excellent work! I have noticed it extraordinary performance in sparse points (32 -> 80%+ in ModelNet40). Recently, I am trying to utilize it to detect objects. However, after grouping the raw points, I need to feed every group of points into en independent PointCNN. However, I had trouble in doing so.

This is because the the input is (Batch_size x Group_num x Points_of_Each_Group x XYZ), but the input of PointCNN is (Batch_size x Points_num x XYZ). More specifically, my input is (128 x 64 x 64 x 3) whose dimension is 4, but the dimension of PointCNN's expected input is 3.

I don't want to feed one group one by one, for it is inefficient so I am wondering whether the code of PointCNN (pointcnn.py for example) could be used in parallel computing of different group. I think it is plausible only by changing a few lines of code. Really hope you can help me solve this problem.

Thank you in advance !!

clarify on input variables in xconv()

Hi, authors,

In funtion xconv() , there are many input variables. Could you help to clarify their meaning? According to PointCNN paper. I guess them as follows. Pls help to correct them if there's any mistake:

input: 
    pts: input points, (neighborhood) point matrix
    fts: input point features, (neighborhood point) feature matrix
    qrs: representative points
    K: nunmber of neighboring points around representative point by K-Nearest Neighbors search
    N: number of output points
    P:  ??
    D: dilation rate
    C: channels of output feature, equals to C_2 in paper
    C_pts_fts: channels of input feature, equals to C_delta + C_1 in paper

Moreover, what does "P" means ?

THX!

problem of prepare_multiChannel_seg_data.py about S3DIS

When I run prepare_multiChannel_seg_data.py for S3DIS, I find that list data doesn't match data_num for Area2-Area6.
I don't know whether it will affect the follow-up operation, but I think it would be better if we put line77-80:
data = np.zeros((batch_size, max_point_num, args.channel_num)) data_num = np.zeros((batch_size), dtype=np.int32) label = np.zeros((batch_size), dtype=np.int32) label_seg = np.zeros((batch_size, max_point_num), dtype=np.int32)
behind line 81:
for data_folder, label_folder in folders:
Then the list data will match data_num.

tu_berlin cls missing train&val list

Great Work. I ran most experiments. Smooth

One experiment for tu_berlin classification missing train&val file list after raw data conversion.

problem of dataset preparing

i have one or some CAD image(pcl, ply, obj, pcd....) of objects in real word around me ,how can I transfer it to h5 file ? then input this h5 file to pointCNN, thanks!

The output of file_seg is -1

When I run test_s3dis.sh, I find that in the beginning, the contents of output ".seg" files are normal, which are indexes of max point_seg_probs. But the output ".seg" files corresponding to the files which are later processed are abnormal. The contents of these ".seg" files are all "-1".

And I run test_general_seg.py in "debug" mode of PyCharm for several "for" loops, the output ".seg" files are all normal, but it become "-1" again once using "run" mode of PyCharm for later processed files.

In test_general_seg.py, line 136-153:

            for idx in range(sample_num*batch_size):
                point_idx = indices_shuffle[idx]
                point_seg_probs = seg_probs_2d[idx, :]
                prob = np.amax(point_seg_probs)
                seg_idx = np.argmax(point_seg_probs)
                if prob > predictions[point_idx][1]:

                    predictions[point_idx] = [seg_idx, prob, point_seg_probs]

            labels = []
            with open(output_filelist[batch_idx], 'w') as file_seg:
                for seg_idx, prob, probs in predictions:

                    file_seg.write(str(int(seg_idx)))

                    file_seg.write("\n")

                    labels.append(seg_idx)

The seg_idx write to file_seg is np.argmax(point_seg_probs), which can't be "-1".

I am really confused, could you please help me ?

Problem in test cifar10 data

There is error in test cifar10 when I follow README.md
Please change to

$ python3 ./prepare_cifar10_data.py -f ../../data/cifar10

$ python3 ./prepare_cifar10_data.py -f ../../data/cifar10/cifar-10-batches-py

Scannet train/test size

I downloaded the scannet dataset and run the code scannet extract object. However, there are only 7815 instances in the trainset and 2170 in the testset compared to 9035 and 2060 respectively stated in the PointCNN paper.

Also when i run the conversion code, I faced "index out of range error" when extract ShapeNetCore 55 labbel: ShapeNetCore55 = line_s[11] . I removed the strip function line_s = line.strip().split('\t') and it works but result in inconsistent train/test size.
Can you have a look of this issue. Thank you.

Undefined name 'tnet'

https://github.com/yangyanli/PointCNN/blob/master/pointnetpp_cls/utils/pointnet_util.py#L49

flake8 testing of https://github.com/yangyanli/PointCNN

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./pointnetpp_cls/utils/pointnet_util.py:49:23: F821 undefined name 'tnet'
        grouped_xyz = tnet(grouped_xyz, tnet_spec)
                      ^
1     F821 undefined name 'tnet'

Reasons on using separable convolution

Hi, @yangyanli ,

I found that there are some places which uses separable convolution. What's the purpose on using it rather than conventional convolution operation? I didn't find any explanation on it in PointCNN paper. Is the performance reason or memory cost reason on using separable convolution?

THX!

Why need to multiply extra coefficient in this formula

line 120 in semantic3d_split.py, why need to multiply nvox[0] and nvox[1]?
line120: vidx = vidx[:, 0] + vidx[:, 1] * nvox[0] + vidx[:, 2] * nvox[0] * nvox[1]

I do not understand the downsample method. Does this method have a name or some reference?

Memory Occupation for Shapenet

I tried to train Shapenet on P40 with 20 GB memory but it still needs more memory.
The config is the default one which you provided with batch_size = 24 but the log file told me that the memory is not enough for creating new blocks.
By the way, could you please tell me about your training config?
Thank you!

xconv_params and xdconv_params description.

Could someone write a simple explanation of how xconv_params and xdconv_params affect the training?

problems when training modelnet

when i train the modelnet with tensorflow 1.4, The process halt on and seems not running, should it read h5 file slowly or really does not run, but it has allocate GPU memory already? Message halt on like these as following:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)

Understanding of downsampling in s3dis_split.py

In data_conversions/split_data/s3dis_split.py line 73 and 75:

            vidx = vidx[:, 0] + vidx[:, 1] * nvox[0] + vidx[:, 2] * nvox[0] * nvox[1]

            uvidx, vpidx = np.unique(vidx, return_index=True)

I think the aim of the code is to leave only one point in the same plane while removing other points in the plane. I don't know whether my understand is correct. And I wonder the feature of these planes, such as their shapes and the aim that you chose nvox[0], nvox[0] * nvox[1] as the coefficients.

I would be grateful if you could explain it to me!

Classification ModelNet40 clip_value_min and clip_value_max

HI,

I trying to train and evaluate for the ModelNet40 data but tensorflow seem to be coming up with the following error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: clip_value_min and clip_value_max must be either of the same shape as input, or a scalar. input shape: [128,1024,3]clip_value_min shape: [1]clip_value_max shape: [1] 

Any advice?

Thanks,

Thibault

segmentation on LIDAR point cloud dataset

Can this architecture be used for segmentation on large areas of LIDAR point clouds? As an example, for the Semantic3D dataset.

problem of using eval_s3dis.py

When using eval_s3dis.py, the output shows that overall accuracy is 1.0 and IoU is 1.0.
I guess there is something wrong with the code in line25:
pred_data_label_filenames = gt_label_filenames
right?

Thanks!

The problem about environment

/usr/local/lib/python3.5/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "../train_val_seg.py", line 271, in
main()
File "../train_val_seg.py", line 126, in main
t_1_acc_op, t_1_acc_update_op = tf.metrics.precision_at_k(labels_sampled, logits, 1)
AttributeError: module 'tensorflow.python.ops.metrics' has no attribute 'precision_at_k'

And I find that the API 'precision_at_k' only exists in tensorflow 1.5 and above, how could you test the code in the environment tensorflow 1.4?

Thank you very much!

module 'shapenet_x8_2048_fps' has no attribute 'step_val'

Hi ,thanks for your share!
When I try to train the shapenet_x8_2048_fps, I accept an error :
File "../train_val_seg.py", line 51, in main
step_val = setting.step_val
AttributeError: module 'shapenet_x8_2048_fps' has no attribute 'step_val'
I checked shapenet_x8_2048_fps.py and found it did not contain "step_val ". What value should be set for step_val ?
Awaiting for your reply .Thank you very much!

High memeory occupation in training for segmentation

Hi ,thanks for your share!
Because my GPU doesn't satisfy the requirement of TensorFlow with GPU, I just use the CPU to do the training. It does work, but I find it occupy 100%CPU and 98%RAM (CPU: i5, RAM: 8G) when I train the shapenet_x8_2048_fps for segmentation, even though I just load only one .h5 file ( which is only 162Mb).
There is anything wrong with my training? Or everything is OK?
Waiting for your reply. Thank you very much!

About furthest point sampling

I want to learn about the algorithm furthest point sampling which is used in your paper. Do you have any recommended reference?

about the fine tune?

1.I have tried your framework,it did works well.But now i want use it to detect fruits,such as orange.How can i fine tune the net to achieve my target?Should i make 3D datasets which just includes orange?Just like make VOC2007 datasets using 2D conv?I plan to use scannet datasets form to make my own datasets,is this proper?
2.你好，我试用了你们的网络，用的是scannet数据集跑的，效果确实不错。我现在想拿它来识别柑橘，实现这种目标需要调优网络吧，那么数据集怎么做呢，自己仿造scannet数据集就可以吗，之前自制VOC2007数据集已经是轻车熟路了，但是3D数据集还不是太了解，感觉这个网络的hyper-para比2D的网络要多啊，会不会很难调？期待您的回复！