loicland / superpoint_graph

Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs

Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs

⚠️ This repo is no longer maintained! Please check out our brand new SuperPoint Transformer, which does everything better! ⚠️

This is the official PyTorch implementation of the papers:

Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs

by Loic Landrieu and Martin Simonovski (CVPR2018),

and

Point Cloud Oversegmentation with Graph-Structured Deep Metric Learning.

by Loic Landrieu and Mohamed Boussaha (CVPR2019),

Code structure

• ./partition/* - Partition code (geometric partitioning and superpoint graph construction using handcrafted features)
• ./supervized_partition/* - Supervized partition code (partitioning with learned features)
• ./learning/* - Learning code (superpoint embedding and contextual segmentation).

To switch to the stable branch with only SPG, switch to release.

Disclaimer

Our partition method is inherently stochastic. Hence, even if we provide the trained weights, it is possible that the results that you obtain differ slightly from the ones presented in the paper.

Requirements

0. Download current version of the repository. We recommend using the --recurse-submodules option to make sure the cut pursuit module used in /partition is downloaded in the process. Wether you did not used the following command, please, refer to point 4:

git clone --recurse-submodules https://github.com/loicland/superpoint_graph

1. Install PyTorch and torchnet.

pip install git+https://github.com/pytorch/tnt.git@master

2. Install additional Python packages:

pip install future igraph tqdm transforms3d pynvrtc fastrlock cupy h5py sklearn plyfile scipy pandas

3. Install Boost (1.63.0 or newer) and Eigen3, in Conda:

conda install -c anaconda boost; conda install -c omnia eigen3; conda install eigen; conda install -c r libiconv

4. Make sure that cut pursuit was downloaded. Otherwise, clone this repository or add it as a submodule in /partition:

cd partition
git submodule init
git submodule update --remote cut-pursuit

5. Compile the libply_c and libcp libraries:

CONDAENV=YOUR_CONDA_ENVIRONMENT_LOCATION
cd partition/ply_c
cmake . -DPYTHON_LIBRARY=$CONDAENV/lib/libpython3.6m.so -DPYTHON_INCLUDE_DIR=$CONDAENV/include/python3.6m -DBOOST_INCLUDEDIR=$CONDAENV/include -DEIGEN3_INCLUDE_DIR=$CONDAENV/include/eigen3
make
cd ..
cd cut-pursuit
mkdir build
cd build
cmake .. -DPYTHON_LIBRARY=$CONDAENV/lib/libpython3.6m.so -DPYTHON_INCLUDE_DIR=$CONDAENV/include/python3.6m -DBOOST_INCLUDEDIR=$CONDAENV/include -DEIGEN3_INCLUDE_DIR=$CONDAENV/include/eigen3
make

6. (optional) Install Pytorch Geometric

The code was tested on Ubuntu 14 and 16 with Python 3.5 to 3.8 and PyTorch 0.2 to 1.3.

Troubleshooting

Common sources of errors and how to fix them:

• $CONDAENV is not well defined : define it or replace $CONDAENV by the absolute path of your conda environment (find it with locate anaconda)
• anaconda uses a different version of python than 3.6m : adapt it in the command. Find which version of python conda is using with locate anaconda3/lib/libpython
• you are using boost 1.62 or older: update it
• cut pursuit did not download: manually clone it in the partition folder or add it as a submodule as proposed in the requirements, point 4.
• error in make: 'numpy/ndarrayobject.h' file not found: set symbolic link to python site-package with sudo ln -s $CONDAENV/lib/python3.7/site-packages/numpy/core/include/numpy $CONDAENV/include/numpy

Running the code

To run our code or retrain from scratch on different datasets, see the corresponding readme files. Currently supported dataset are as follow:

Dataset	handcrafted partition	learned partition
S3DIS	yes	yes
Semantic3D	yes	to come soon
vKITTI3D	no	yes
ScanNet	to come soon	to come soon

To use pytorch-geometric graph convolutions instead of our own, use the option --use_pyg 1 in ./learning/main.py. Their code is more stable and just as fast. Otherwise, use --use_pyg 0

Evaluation

To evaluate quantitatively a trained model, use (for S3DIS and vKITTI3D only):

python learning/evaluate.py --dataset s3dis --odir results/s3dis/best --cvfold 123456

To visualize the results and all intermediary steps, use the visualize function in partition (for S3DIS, vKITTI3D,a nd Semantic3D). For example:

python partition/visualize.py --dataset s3dis --ROOT_PATH $S3DIR_DIR --res_file results/s3dis/pretrained/cv1/predictions_test --file_path Area_1/conferenceRoom_1 --output_type igfpres

output_type defined as such:

• 'i' = input rgb point cloud
• 'g' = ground truth (if available), with the predefined class to color mapping
• 'f' = geometric feature with color code: red = linearity, green = planarity, blue = verticality
• 'p' = partition, with a random color for each superpoint
• 'r' = result cloud, with the predefined class to color mapping
• 'e' = error cloud, with green/red hue for correct/faulty prediction
• 's' = superedge structure of the superpoint (toggle wireframe on meshlab to view it)

Add option --upsample 1 if you want the prediction file to be on the original, unpruned data (long).

Other data sets

You can apply SPG on your own data set with minimal changes:

• adapt references to custom_dataset in /partition/partition.py
• you will need to create the function read_custom_format in /partition/provider.py which outputs xyz and rgb values, as well as semantic labels if available (already implemented for ply and las files)
• adapt the template function /learning/custom_dataset.py to your achitecture and design choices
• adapt references to custom_dataset in /learning/main.py
• add your data set colormap to get_color_from_label in /partition/provider.py
• adapt line 212 of learning/spg.py to reflect the missing or extra point features
• change --model_config to gru_10,f_K with K as the number of classes in your dataset, or gru_10_0,f_K to use matrix edge filters instead of vectors (only use matrices when your data set is quite large, and with many different point clouds, like S3DIS).

Datasets without RGB

If your data does not have RGB values you can easily use SPG. You will need to follow the instructions in partition/partition.ply regarding the pruning. You will need to adapt the /learning/custom_dataset.py file so that it does not refer ro RGB values. You should absolutely not use a model pretrained on values with RGB. instead, retrain a model from scratch using the --pc_attribs xyzelpsv option to remove RGB from the shape embedding input.

Citation

If you use the semantic segmentation module (code in /learning), please cite:
Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs, Loic Landrieu and Martin Simonovski, CVPR, 2018.

If you use the learned partition module (code in /supervized_partition), please cite:
Point Cloud Oversegmentation with Graph-Structured Deep Metric Learning, Loic Landrieu and Mohamed Boussaha CVPR, 2019.

To refer to the handcrafted partition (code in /partition) step specifically, refer to:
Weakly Supervised Segmentation-Aided Classification of Urban Scenes from 3D LiDAR Point Clouds, Stéphane Guinard and Loic Landrieu. ISPRS Workshop, 2017.

To refer to the L0-cut pursuit algorithm (code in github.com/loicland/cut-pursuit) specifically, refer to:
Cut Pursuit: Fast Algorithms to Learn Piecewise Constant Functions on General Weighted Graphs, Loic Landrieu and Guillaume Obozinski, SIAM Journal on Imaging Sciences, 2017

To refer to pytorch geometric implementation, see their bibtex in their repo.

superpoint_graph's Issues

Do you use rgb to generate features during preprocessing?

I want to train without rgb and I succeed.
I want to know that if you use rgb to generate other features when preprocessing the original data? Because if you use rgb, maybe I need to re-generate the spg and feature files. If you do not use rgb during preprocessing, I can just omit the rgb value and train.

Having some issues with running on a custom dataset

I'm trying to just get things working with a custom dataset right now, one that has no RGB values. The amount of data is small (if that matters), and so I'm not really looking for accuracy, just getting segmentation/training/testing working.

I've attempted to follow the steps listed for both custom and no-RGB data, but I'm still having an issue I'm uncertain of. Partitioning goes fine, reorganizing goes fine, but the training step fails with the following -

Traceback (most recent call last):
File "learning/main.py", line 383, in
main()
File "learning/main.py", line 286, in main
acc, loss, oacc, avg_iou = train()
File "learning/main.py", line 183, in train
embeddings = ptnCloudEmbedder.run(model, *clouds_data)
File "/home/ubuntu/dev/spgraph/learning/pointnet.py", line 135, in run_full_monger
out = model.ptn(Variable(clouds, volatile=True), Variable(clouds_global, volatile=True))
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/ubuntu/dev/spgraph/learning/pointnet.py", line 96, in forward
input = self.convs(input)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 176, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight[64, 8, 1], so expected input[3, 5, 128] to have 8 channels, but got 5 channels instead

Any suggestion as to what I've missed/am doing wrong?

STNkD final bias not trainable?

Hi,
in

superpoint_graph/learning/pointnet.py

Line 51 in 518fc08

return input.view(-1,self.eye.size(1),self.eye.size(2)) + Variable(self.eye)

you are making the bias for the projection an Identity matrix and not trainable?

In the original pointnet implementation, they make this a bias term (https://github.com/charlesq34/pointnet/blob/d64d2398e55b24f69e95ecb549ff7d4581ffc21e/models/transform_nets.py#L49), which is trainable.

The STN is actually supposed to output a rotation matrix, so in my reasoning, the bias term should be trainable.

Do you make it untrainable on purpose, and why?

Also, why the projection are initialized as all 0s. In deep learning courses, it's not encouraged to set the weights as the same value.

Inference error in line 336, in resume model.load_state_dict(checkpoint['state_dict']

Hi,

I wanted to do inference on semantic_3d dataset. I downloaded the pretrained model from web and remaed to model.pth.tar.
Kindly help, But it landed error throwing exception:

Traceback (most recent call last):
File "/usr/share/pycharm/helpers/pydev/pydevd.py", line 1599, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/usr/share/pycharm/helpers/pydev/pydevd.py", line 1026, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/usr/share/pycharm/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/main.py", line 387, in
main()
File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/main.py", line 144, in main
model, optimizer, stats = resume(args, dbinfo)
File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/main.py", line 336, in resume
model.load_state_dict(checkpoint['state_dict'])
File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Module:
Unexpected running stats buffer(s) "ecc.0._cell.ini.running_mean" and "ecc.0._cell.ini.running_var" for InstanceNorm1d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm1d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm1d to enable them. See the documentation of InstanceNorm1d for details.
Unexpected running stats buffer(s) "ecc.0._cell.inh.running_mean" and "ecc.0._cell.inh.running_var" for InstanceNorm1d with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because InstanceNorm1d does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in InstanceNorm1d to enable them. See the documentation of InstanceNorm1d for details.

Process finished with exit code 1

Confusion matrix on custom dataset

Hey Loic,

I am trying to train SPG on a 5-class labeled dataset (basically synthia: 360-degree-depthcams -> pointcloud)

I have the data in .ply format and as far as I can tell, loading of PC and labels as well as partitioning and SPG generation works. However, I am getting this error during training:

Epoch 0/500 (results/synthia_model):
0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "learning/main.py", line 383, in
main()
File "learning/main.py", line 286, in main
acc, loss, oacc, avg_iou = train()
File "learning/main.py", line 200, in train
confusion_matrix.count_predicted_batch(tvec_cpu, np.argmax(o_cpu,1))
File "/home/max/synthia_spg/learning/metrics.py", line 18, in count_predicted_batch
self.confusion_matrix[:,predicted[i]] += ground_truth_vec[i,:]
IndexError: index 7 is out of bounds for axis 1 with size 5

I checked the confusion matrix and it is generated as 5x5 with zeros so that seems to work fine (it reads the 5 from custom_dataset.py).

1. The tvec_cpu matrix in the line mentioned above seems to be an one-hot representation of my labels, however instead of all 0 but one =1 the "hot" entry is some random integer. Is that correct?

2. The o_cpu matrix is a matrix of same row count as tvec_cpu. However, it has a column count of 7 (0-6) and contains a bunch of values, mostly +-0.XXXXX - can you explain what this exactly is?

3. I think the problem lies somewhere within these passed arguments and the dimensionality of o_cpu compared to tvec_cpu.

I know this information is vague but any help is greatly appreciated since I am a little stuck here :D

Best,
Max

Issues with S3DIS Dataset

So I'm attempting to replicate the S3DIS dataset results. I've endeavored to set things up to where they are listed in the readme, with the only things I know to be different are I'm using Python 3.6.4, latest Pytorch (since readme says to pull off of master), and Boost 1.67 (since readme says 1.63 or newer).

In running against the unzipped dataset, I'm getting, on Area 1 -

Voxelization into 199 x 206 x 116 grid
Reduced from 1535040 to 188713 points (12.29%)
95% done
Traceback (most recent call last):
File "partition/partition.py", line 155, in
write_features(fea_file, geof, xyz, rgb, graph_nn, labels)
File "/home/ubuntu/dev/spgraph/partition/provider.py", line 344, in write_features
data_file.create_dataset('rgb', data=rgb, dtype='uint8')
File "/home/ubuntu/anaconda3/envs/py360/lib/python3.6/site-packages/h5py/_hl/group.py", line 109, in create_dataset
self[name] = dset
File "/home/ubuntu/anaconda3/envs/py360/lib/python3.6/site-packages/h5py/_hl/group.py", line 277, in setitem
h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)

I went ahead and stuck the call in h50.link in a try/except block just to allow things to move on (rather than trying to diagnose what was going on, just to see if maybe it didn't matter), and that allowed things to progress; however, then on Area 3 I got -

Voxelization into 53 x 549 x 92 grid
Reduced from 178968 to 22650 points (12.65%)
88% done
computing the superpoint graph...
minimal partition...
L0-CUT PURSUIT WITH L2 FIDELITY
PARAMETERIZATION = FAST
Graph 22652 vertices and 543600 edges and observation of dimension 7
Iteration 1 - 58 components - Saturation 0.0 % - Quadratic Energy 46.465 % - Timer 0.226557
Iteration 2 - 194 components - Saturation 0.7 % - Quadratic Energy 28.743 % - Timer 0.493281
Iteration 3 - 350 components - Saturation 4.3 % - Quadratic Energy 23.606 % - Timer 0.78682
Iteration 4 - 489 components - Saturation 23.6 % - Quadratic Energy 21.411 % - Timer 1.08708
Iteration 5 - 571 components - Saturation 44.1 % - Quadratic Energy 20.574 % - Timer 1.33033
Max number of iteration reached
computation of the SPG...
Timer : 132.3 / 621.9 / 998.5
2 / 23---> hallway_2
creating the feature file...
Traceback (most recent call last):
File "partition/partition.py", line 123, in
xyz, rgb, labels = read_s3dis_format(data_file)
File "/home/ubuntu/dev/spgraph/partition/provider.py", line 162, in read_s3dis_format
nn = NearestNeighbors(1, algorithm='kd_tree').fit(xyz)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/neighbors/base.py", line 803, in fit
return self._fit(X)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/neighbors/base.py", line 204, in _fit
X = check_array(X, accept_sparse='csr')
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 453, in check_array
_assert_all_finite(array)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 44, in _assert_all_finite
" or a value too large for %r." % X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

Any help on what might be different would be appreciated.

Superedge features with Naan values

Hello !

Disclaimer : this is not really a question, rather a warning for an issue I came accross when applying the s3dis-pretrained model on my own data for inference and that someone else might encounter.

It so happened that preprocessing.StandardScaler().fit(edge_feats) in learning/spg.py would throw a "Nan or infinite or non-float32" error when trying to scale the superedge features.

After investigating a little, I found that this was caused by the fact that the 3rd eigenvalue of the PCA of superpoints in compute_sp_graph() of partition/graphs.py may be slightly negative, due to numerical instability for skewed superpoint shapes I presume. This in return may affect the computation of the sp_volume feature at graph["sp_volume"][i_com] = np.sqrt(ev[0] * ev[1] * ev[2] + 1e-10). If the product of the eigenvalues is negative enough, the sqrt returns a Nan without throwing an error.

To overcome this, I just changed ev = -np.sort(-ev[0]) into ev = -np.sort(-np.abs(ev[0])), to ensure all PCA eigenvalues are positive.

Windows Support?

Hello and thank you for making this code available.

Is it possible to use this code on a Windows machine or is it for Linux only?

Thanks.

Inference on custom indoor dataset with S3DIS-trained model : fnet weights don't agree

Hello,

First a big thank you for sharing this impressive project and working hard on making it reusable!

I have been trying to recover your trained S3DIS best models and test them for inference on a dataset of mine. Hence, after adapting the recommended files in /partition and /learning, I first successfully run the unsupervized partition on my custom_indoor_data with :

python partition/partition.py --dataset custom_indoor --ROOT_PATH $INDOOR_DATA --voxel_width 0.03 --reg_strength 0.03

Then I runI the SPG segmentation with :

for FOLD in 1 2 3 4 5 6; do CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset custom_indoor --CUSTOM_INDOOR_PATH $INDOOR_DATA --cvfold $FOLD --epochs -1 --lr_steps '[275,320]' --test_nth_epoch 50 --model_config 'gru_10_0,f_13' --ptn_nfeat_stn 14 --nworkers 2 --odir "models/s3dis/cv${FOLD}" --resume RESUME; done

However, when loading the weights from the checkpoint, there seems to be a disagreement in the output dimension of the the 7th linear layer of the fnet :

RuntimeError: While copying the parameter named ecc.0._fnet.7.weight, whose dimensions in the model are torch.Size([32, 64]) and whose dimensions in the checkpoint are torch.Size([1024, 64])

Would you have an idea of what may be causing this ? Have I poorly parameterized my inference ? Are the checkpoint weights provided not the final ones ?

Thanks in advance for the tips !

Issues compiling ply_c library

Hi,
Thank you so much for your wonderful work. I have struggled for weeks on trying to compile the ply_c and cut-pursuit libraries. It seems that anaconda is not allowing cmake to find the real paths both libraries need. How do you go around this? Have you experienced similar issues? Here is the output of cmake:

cmake . -DPYTHON_LIBRARY=$CONDAENV/lib/libpython3.6m.so -DPYTHON_INCLUDE_DIR=$CONDAENV/include/python3.6m -DBOOST_INCLUDEDIR=$CONDAENV/include -DEIGEN3_INCLUDE_DIR=$CONDAENV/include/eigen3
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning at /usr/share/cmake-3.5/Modules/FindBoost.cmake:725 (message):
  Imported targets not available for Boost version 106501
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:763 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:1332 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:21 (find_package)


CMake Warning at /usr/share/cmake-3.5/Modules/FindBoost.cmake:725 (message):
  Imported targets not available for Boost version 106501
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:763 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:1332 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:21 (find_package)


-- Boost version: 1.65.1
-- Found the following Boost libraries:
--   graph
--   numpy
-- Found PythonLibs: /lib/libpython3.6m.so  
PYTHON LIBRARIES ARE /lib/libpython3.6m.so
-- Configuring done
CMake Warning at src/CMakeLists.txt:3 (add_library):
  Cannot generate a safe linker search path for target cp because files in
  some directories may conflict with libraries in implicit directories:

    link library [libpython3.6m.so] in /lib may be hidden by files in:
      /home/siquike/anaconda3/lib

  Some of these libraries may not be found correctly.


CMake Warning at src/CMakeLists.txt:3 (add_library):
  Cannot generate a safe runtime search path for target cp because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libpython3.6m.so] in /lib may be hidden by files in:
      /home/siquike/anaconda3/lib

  Some of these libraries may not be found correctly.


-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    EIGEN3_INCLUDE_DIR

Variable number of points for pointnet features for each superpoint - detection ideas.

Hey Loic,

I just spent a few hours with your paper. Really enjoyed it. I'm starting a new project on instance-level detection of trees in lidar + orthophotos (some public data here). I was already moving in the direction of superpoints with pointnet for feature extraction, but I don't quite understand how/when the subsampling happens.

In section 3.3, i see

In our case, input shapes are geometrically simple ob- jects, which can be reliably represented by a small amount of points and embedded by a rather compact PointNet. This is important to limit the memory needed when evaluating many superpoints on current GPUs. In particular, we sub- sample superpoints on-the-fly down to np = 128 points to maintain efficient computation in batches and facilitate data augmentation. Superpoints of less than np points are sampled with replacement, which in principle does not affect the evaluation of PointNet due to its max-pooling.

Are the same 128 points sampled every run? How are they chosen? Do you think it will matter? Sensitivity to the number?

What I did was first a voxelgrid is overlaid on the cloud to regularize the point cloud. This enforces a general downsampling and tries to keep the overall structure. From there, I find the distance to the closest neighbor for each point. I use this vector to weight a random sampling of the array, such that points with close neighbors are less likely to be sampled. This is a computationally expensive task. If there are fewer points in a scene than the fixed number, I sample with replacement to create duplicates. I haven't read anywhere what effect this duplication has, but I can see you did it too.

Thanks. If you have any thoughts on turning these approaches in detection, rather than semantic segmentation, let me know. My basic thought is

1. Unsupervised classification to create superpoint objects
2. Classify intro tree/not tree
3. Iterative graph cut, where during each cut the resulting objects are compared to a classification network on individual trees. Stop cutting when classification probability decreases.

Getting all-zero labels for reduced test set with downloaded pre-trained model

Hello,

I am trying to get some segmentation results on the Semantic3D reduced test set.

I have downloaded the pre-trained model from the following link, renamed it model.pth.tar and placed it under /results/sema3d/trainval_best
http://imagine.enpc.fr/~simonovm/largescale/model_sema3d_trainval.pth.tar

I have run partition.py on the reduced test set.

I then run the following two commands:

python learning/sema3d_dataset.py --SEMA3D_PATH $SEMA3D_DIR

CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset sema3d --SEMA3D_PATH $SEMA3D_DIR --db_test_name testred --db_train_name trainval --epochs 500 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11 --nworkers 2 --odir "results/sema3d/trainval_best" --resume RESUME

After the second one, I get the following warning:

100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.69s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.64s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:15<00:00, 3.83s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.71s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.69s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.74s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:15<00:00, 3.79s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.73s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.75s/it]
100%|█████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.73s/it]
~/repositories/SuperPointGraphs/learning/metrics.py:66: RuntimeWarning: invalid value encountered in true_divide
return sum(values) / class_seen
-> Multisample 10: Test accuracy: 100.0, Test oAcc: 0.0, Test avgIoU: nan, Test mAcc: 0.0

After this point, when I write the labels, I see that they are all zero, and the resulting prediction point cloud has a single grey color.

Can you see what I'm doing wrong in this process?

Thanks for your help.

Nadaraya-Watson estimation

Hi, loicland：
have you released the code about Nadaraya-Watson estimation for interpolating labels of full point cloud ? thks

Questions about partition on a large colored point cloud

Hi Loic,

When I tried to run partition on a large scale colored point cloud (contains 5000000 points), I got the following error:
/home/ubuntu/capstone/superpoint_graph/partition/provider.py:357: UserWarning: genfromtxt: Empty input file: "/home/ubuntu/capstone/semantic3d/data/test_full/colored.txt" , skip_header=i_rows) Traceback (most recent call last): File "partition/partition_Semantic3D.py", line 93, in <module> xyz, rgb = prune(data_file, args.ver_batch, args.voxel_width) File "/home/ubuntu/capstone/superpoint_graph/partition/provider.py", line 361, in prune xyz_full = np.array(vertices[:, 0:3], dtype='float32') IndexError: too many indices for array
The command I run here is:partition/partition_Semantic3D.py --SEMA3D_PATH $SEMA3D_DIR.You can find my file at:

I also tried to reduce the file size (I created another txt file and copy the first 691892 lines of my original colored point cloud to this new file, you can find it at:) and re-run partition with the same command. This time the error disappeared, but the number of point didn't reduced at all (the log is Reduced from 691892 to 691892 points (37.92%)). I remembered that when I run partition on a point cloud without color (you can find it at ), the point number drastically reduced (the log is Reduced from 635137 to 289 points (0.04%)). Could you please tell me reason of it?

Many Thanks,
Yongchi

Pytorch 0.4 (I believe) prevents results from being written

On line 299 in learning/main.py, a call is made to json.dump(), which is causing an error of "TypeError: Object of type 'Tensor' is not JSON serializable". A little searching and I've seen one or two other errors indicating that the upgrade to Pytorch 0.4 may be responsible.

Calling str() on the tensor first will cause it to write out in a readable format. If that's the only goal (I don't know what the original call to json.dump did, other than, obviously, represented it as JSON. I don't know if it's the same representation), I can make a pull request; if there are other expectations on that output (it may show up when I get to those steps) I can take a stab at them, but it may warrant additional attention.

Also, just a warning, but future proofing (so either lock down the Pytorch version or fix; easy fix), getting a few deprecation warnings related to accessing loss.data[0] -

learning/main.py:196: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
loss_meter.add(loss.data[0])
learning/main.py:202: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
logging.debug('Batch loss %f, Loader time %f ms, Trainer time %f ms.', loss.data[0], t_loader, t_trainer)

I can put in a pull request for that fix as well, though I'm not sure if those changes would be breaking with earlier Pytorch versions (migration guide is here, didn't look in that detail) - https://pytorch.org/2018/04/22/0_4_0-migration-guide.html

On that note, 'volatile' is also deprecated. So might just be easier to lock the Pytorch version (maybe an environment.txt file?)

Can't file .hpy file

Was having an issue finding a predictions_testred.h5 hpy file: realized a new trainval_best folder is generated even though the command: python partition/write_Semantic3d.py --SEMA3D_PATH $SEMA3D_DIR --odir "results/sema3d/best" --db_test_name testred specifies a different path in the folder 'best'

Found this command to work which implies predictions_testred.h5 is in trainval_best folder and not best folder. Not sure if this is a problem other people are facing? If anyone has this issue as well this is the solution I found.
amanda@amanda-Precision-Tower-5810:~/superpoint_graph$ python partition/write_Semantic3d.py --SEMA3D_PATH /home/amanda/Semantic3D/ --odir "results/sema3d/trainval_best" --db_test_name testred

OSError: Unable to open file (unable to open file: name = './results/sema3d/best/predictions_testred.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

the accuracy and IoU for training dataset of Semantic3d

Hi, @loicland , thanks for your work, and when I want to refine the results of your model ,I generate the prediction results of training dataset of semantic3d by using your offered pretrained model, i.e., I use the modified main.py with cmd: --db_test_name train, I found that the prediction accuracy is much lower than your results on the benchmark.
Test accuracy: 85.96092934693056, Test oAcc: 0.6773388512039751, Test avgIoU: 0.45078531031139096, Test mAcc: 0.63300907

I wonder why ? Thanks a lot.

About the label of the test data in Semantic3d Dataset

Hello. I have a question about semantic3d dataset.

I find that the dataset does not provide labels of the test data. I want to ask if we can only upload the label
that we predict to the website to check the accuracy. Can we get the label of the test data?

Thank you very much!

Partitioning Semantic 3d dataset - Performance

I am currently executing partition step for Semantic 3d dataset but it is taking a lot of time. How much time approximately does this script takes to partition all test and training point clouds of Semantic 3d dataset?
The script has been executing for 2 days approx and still it is partitioning first pointcloud (sg28_Station2_rgb_intensity-reduced of Reduced Test set) . My machine has 16 GB Ram and the code is utilizing swap memory too. I have increased voxel_width to 0.5 and ver_batch decreased to 500. Am I doing something wrong?
Also can you provide me the partitioned output files i.e. .h5 format for Semantic 3d dataset?

Question about the label

Hello,
I am trying to train a model on my own dataset. The label in my dataset are 0 and 1. When I create feature.h5 and super_point_graph.h5, It become all-zeros. I am sure of the label while reading correct. It seems that something wrong in "libply_c.prune" because the input label of this function is correct but the output(labels_sub) maybe wrong(col 1 all one, col 2 & 3 all zero)

About the grammar of pytorch

I see that you use model.ecc and model.ptn during the model creating. I do not see such usage of nn.Module() before. I want to ask why can add .ecc and .ptn after variable 'model'. Is there any API or explanation for this grammar?

Thank you very much!

Label computation for SD3IS dataset

Hi,
Thanks a lot for the code release!

When reading the S3DIS data, the code loads one 'object' at a time, and for each 'object vertex', assigns a label to its closest 'room vertex'. Should the nearest neighbor search not be done in the other direction i.e. for each 'room vertex', its label should be same as that of the closest 'object vertex' (across all objects)?

Thanks!

Partitioning taking very long

Hello,
Was wondering if people can provide me with how long the partition step takes for you. When I started using this code: it didn't take very long (minutes), but now it takes 2+ hours tops (don't know if this is a memory issue?). This is just with the base Semantic3D training.

I also recently modified the training to take in an additional 1.8GB of additional data and now it takes more than 24 hours to partition. I lowered the batch to 1000000 but no resolve.

How long does partitioning take for you with the original data? With additional training data you may have added in addition to the Semantic3D one?

These are the parameters I'm using:

--voxel_width 0.01 --reg_strength 0.8 --ver_batch 1000000

here is my RAM use in GB:
free -g
total used free shared buff/cache available
Mem: 125 24 52 0 48 100
Swap: 127 0 127

If the intensity value in SEMANTIC3D dataset is used or not?

First I want to ask that if you use the intensity value in your code as I do not find you use it.

Secondly, I want to ask you if you know the meaning of the negative intensity value in semantic3d dataset. As far as I know, the intensity of LIDAR is usually positive as I only use the Velodye LIDAR.

Thank you very much!

libcp library installation problem (`boost_numpy` not found)

I get the following error when trying to compile the libcp library:

Could not find the following Boost libraries:

          boost_numpy

Some (but not all) of the required Boost libraries were found.  You may
  need to install these additional Boost libraries.  Alternatively, set
  BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
  to the location of Boost.
Call Stack (most recent call first):
  CMakeLists.txt:17 (find_package)

I installed the Boost.Python library (from https://github.com/boostorg/python) but I get the same error.
Anyone with the same problem?

name 'libply_c' is not definded

After make smooth, use python partition_Semantic3D.py --SEMA3D $SEMAN3D_DIR

Get this error

=================
test_reduced/

1 / 14---> sg28_station2
creating the feature file...
Traceback (most recent call last):
File "partition_Semantic3D.py", line 92, in
xyz, rgb = prune(data_file, args.ver_batch, args.voxel_width)
File "/mnt/diskb/dixinhan/Deep_3D/Pointnet/superpoint_graph/partition/provider.py", line 357, in prune
xyz_sub, rgb_sub, l = libply_c.prune(xyz_full, voxel_width
NameError: name 'libply_c' is not defined

Error in training AttributeError: 'Program' object has no attribute '_program @ acc, loss, oacc, avg_iou = train() in learning/main.py"

Hi,
Currently using 0.3.1 pytorch
a) Did partition of semantic 3d data using
python partition/partition.py --dataset sema3d --ROOT_PATH /home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d --voxel_width 0.05 --reg_strength 0.8 --ver_batch 5000000
b) reorganize point clouds into superpoints by
python learning/sema3d_dataset.py --SEMA3D_PATH /home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d

But got the error below: Kindly help

Epoch 0/500 (/home/administrator/Documents.orig/Kumar_FileSystem/results/sema3d/trainval_best):
0%| | 0/7 [00:00<?, ?it/s]Traceback (most recent call last):
File "learning/main.py", line 387, in
main()
File "learning/main.py", line 286, in main
acc, loss, oacc, avg_iou = train()
File "learning/main.py", line 184, in train
outputs = model.ecc(embeddings)
File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/graphnet.py", line 97, in forward
input = module(input)
File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/modules.py", line 54, in forward
input = ecc.GraphConvFunction(nc, nc, idxn, idxe, degs, degs_gpu, self._edge_mem_limit)(hx, weights)
File "/home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/ecc/GraphConvModule.py", line 67, in forward
cuda_kernels.conv_aggregate_fw(output.narrow(0,startd,numd), products.view(-1,self._out_channels), self._degs_gpu.narrow(0,startd,numd))
File "/home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 120, in conv_aggregate_fw
function, stream = get_kernel_func('conv_aggregate_fw_kernel_v2', conv_aggregate_fw_kernel_v2(), get_dtype(src))
File "/home/administrator/Documents.orig/FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 33, in get_kernel_func
prog = Program(ksrc.encode('utf-8'), (kname+dtype+'.cu').encode('utf-8'))
File "/home/administrator/anaconda3/lib/python3.6/site-packages/pynvrtc/compiler.py", line 52, in init
include_names)
File "/home/administrator/anaconda3/lib/python3.6/site-packages/pynvrtc/interface.py", line 200, in nvrtcCreateProgram
c_char_p(encode_str(src)), c_char_p(encode_str(name)),
File "/home/administrator/anaconda3/lib/python3.6/site-packages/pynvrtc/interface.py", line 54, in encode_str
return s.encode("utf-8")
AttributeError: 'bytes' object has no attribute 'encode'
Exception ignored in: <bound method Program.del of <pynvrtc.compiler.Program object at 0x7f3c4c640c50>>
Traceback (most recent call last):
File "/home/administrator/anaconda3/lib/python3.6/site-packages/pynvrtc/compiler.py", line 56, in del
self._interface.nvrtcDestroyProgram(self._program)
AttributeError: 'Program' object has no attribute '_program

AttributeError: 'bytes' object has no attribute 'encode'

Got error when running learning/main.py, my running environment :
ubuntu 16.04
pytorch 0.3.0 CUDA 9.0
Notice that I'm running with Semantic 3D datasets.

AttributeError: 'bytes' object has no attribute 'encode'
AttributeError: 'Program' object has no attribute '_program'

  Will save to results/sema3d/trainval_best
Total number of parameters: 213772
Module(
  (ecc): GraphNetwork(
    (0): RNNGraphConvModule(
      (_cell): GRUCellEx(32, 32)(ingate layernorm)
      (_fnet): Sequential(
        (0): Linear(in_features=13, out_features=32, bias=True)
        (1): ReLU(inplace)
        (2): Linear(in_features=32, out_features=128, bias=True)
        (3): ReLU(inplace)
        (4): Linear(in_features=128, out_features=64, bias=True)
        (5): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
        (6): ReLU(inplace)
        (7): Linear(in_features=64, out_features=32, bias=False)
      )
    )
    (1): Linear(in_features=352, out_features=8, bias=True)
  )
  (ptn): PointNet(
    (stn): STNkD(
      (convs): Sequential(
        (0): Conv1d(11, 64, kernel_size=(1,), stride=(1,))
        (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
        (2): ReLU(inplace)
        (3): Conv1d(64, 64, kernel_size=(1,), stride=(1,))
        (4): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
        (5): ReLU(inplace)
        (6): Conv1d(64, 128, kernel_size=(1,), stride=(1,))
        (7): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True)
        (8): ReLU(inplace)
      )
      (fcs): Sequential(
        (0): Linear(in_features=128, out_features=128, bias=True)
        (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True)
        (2): ReLU(inplace)
        (3): Linear(in_features=128, out_features=64, bias=True)
        (4): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
        (5): ReLU(inplace)
      )
      (proj): Linear(in_features=64, out_features=4, bias=True)
    )
    (convs): Sequential(
      (0): Conv1d(11, 64, kernel_size=(1,), stride=(1,))
      (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
      (2): ReLU(inplace)
      (3): Conv1d(64, 64, kernel_size=(1,), stride=(1,))
      (4): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
      (5): ReLU(inplace)
      (6): Conv1d(64, 128, kernel_size=(1,), stride=(1,))
      (7): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True)
      (8): ReLU(inplace)
      (9): Conv1d(128, 128, kernel_size=(1,), stride=(1,))
      (10): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True)
      (11): ReLU(inplace)
      (12): Conv1d(128, 256, kernel_size=(1,), stride=(1,))
      (13): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True)
      (14): ReLU(inplace)
    )
    (fcs): Sequential(
      (0): Linear(in_features=257, out_features=256, bias=True)
      (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True)
      (2): ReLU(inplace)
      (3): Linear(in_features=256, out_features=64, bias=True)
      (4): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True)
      (5): ReLU(inplace)
      (6): Linear(in_features=64, out_features=32, bias=True)
    )
  )
)
Epoch 0/500 (results/sema3d/trainval_best):
  0%|                                                                         | 0/7 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/main.py", line 388, in <module>
    main()
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/main.py", line 287, in main
    acc, loss, oacc, avg_iou = train()
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/main.py", line 185, in train
    outputs = model.ecc(embeddings)
  File "/home/xuyan/tensorflow_xuyan/virtual_3.5/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/graphnet.py", line 97, in forward
    input = module(input)
  File "/home/xuyan/tensorflow_xuyan/virtual_3.5/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/modules.py", line 54, in forward
    input = ecc.GraphConvFunction(nc, nc, idxn, idxe, degs, degs_gpu, self._edge_mem_limit)(hx, weights)
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/ecc/GraphConvModule.py", line 67, in forward
    cuda_kernels.conv_aggregate_fw(output.narrow(0,startd,numd), products.view(-1,self._out_channels), self._degs_gpu.narrow(0,startd,numd))
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 120, in conv_aggregate_fw
    function, stream = get_kernel_func('conv_aggregate_fw_kernel_v2', conv_aggregate_fw_kernel_v2(), get_dtype(src))
  File "/home/xuyan/pytorch_xuyan/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 33, in get_kernel_func
    prog = Program(ksrc.encode('utf-8'), (kname+dtype+'.cu').encode('utf-8'))
  File "/home/xuyan/tensorflow_xuyan/virtual_3.5/lib/python3.5/site-packages/pynvrtc/compiler.py", line 52, in __init__
    include_names)
  File "/home/xuyan/tensorflow_xuyan/virtual_3.5/lib/python3.5/site-packages/pynvrtc/interface.py", line 200, in nvrtcCreateProgram
    c_char_p(encode_str(src)), c_char_p(encode_str(name)),
  File "/home/xuyan/tensorflow_xuyan/virtual_3.5/lib/python3.5/site-packages/pynvrtc/interface.py", line 54, in encode_str
    return s.encode("utf-8")
AttributeError: 'bytes' object has no attribute 'encode'
Exception ignored in: <bound method Program.__del__ of <pynvrtc.compiler.Program object at 0x7ff0da9abfd0>>
Traceback (most recent call last):
  File "/home/xuyan/tensorflow_xuyan/virtual_3.5/lib/python3.5/site-packages/pynvrtc/compiler.py", line 56, in __del__
    self._interface.nvrtcDestroyProgram(self._program)
AttributeError: 'Program' object has no attribute '_program'

Can I run you trained networks on a computer without GPU?

Hi!
I am trying to run your trained network with the semantic3D reduced-dataset on my computer. But, I don't have GPU on my computer? I use the command:
_python3.5 learning/main.py --dataset sema3d --SEMA3D_PATH /home/us-wrr/code/superpoint_graph-release/sema3d --db_test_name testred --db_train_name trainval --epochs -1 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11 --nworkers 2 --odir "results/sema3d/trainval_best" --resume RESUME_
and have the error:
loading checkpoint 'results/sema3d/trainval_best/model.pth.tar' THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=51 error=35 : CUDA driver version is insufficient for CUDA runtime version Traceback (most recent call last): File "learning/main.py", line 387, in <module> main() File "learning/main.py", line 144, in main model, optimizer, stats = resume(args, dbinfo) File "learning/main.py", line 328, in resume checkpoint = torch.load(args.resume) File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/serialization.py", line 303, in load return _load(f, map_location, pickle_module) File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/serialization.py", line 469, in _load result = unpickler.load() File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/serialization.py", line 437, in persistent_load data_type(size), location) File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/serialization.py", line 88, in default_restore_location result = fn(storage, location) File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/serialization.py", line 70, in _cuda_deserialize return obj.cuda(device) File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/_utils.py", line 68, in _cuda with torch.cuda.device(device): File "/home/us-wrr/.local/lib/python3.5/site-packages/torch/cuda/__init__.py", line 225, in __enter__ self.prev_idx = torch._C._cuda_getDevice() RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:51
How can I run your trained network on my computer? Or, I need to train the network again?

Questions about labels_sub

Thanks for your work!
Why labels_sub is a N X (Class+1) dimensional matrix. It's a simple question, but I don't really understand it.
thank you!

Apply spg on other dataset

Hi,
thank you for your excellent work!
I'm wondering what's the required dataset format if I want to apply your network on other point cloud dataset?
And if possible, is there any instructions on which part of the code I should modify to make SPG work on other dataset?
Thanks!

Another question is what is your working environment?
Noticing that you use both conda and pip.
Will virtualenv work?

How long does this network need to train on the semantic3d?

I train on the semantic3d on my computer and I surprisedly find that the train process is very fast which is nearly 10 minutes(just approximate, but actually it is very fast). I want to ask if it is normal.

It took a lot of time to preprocess the dataset such as generating the feature. But training is very fast.

File "/home/administrator/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py", line 931, in __iter__ for obj in iterable:

I want to train on semantic3d dataset. I followed the procedure to partition first and then train
After executing train "CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset sema3d --SEMA3D_PATH /home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d --db_test_name testred --db_train_name trainval --epochs 500 --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11 --nworkers 2 --odir "/home/administrator/Documents.orig/Kumar_FileSystem/results/sema3d/trainval_best""

it started throwing an error below: Kindly help

Epoch 0/500 (/home/administrator/Documents.orig/Kumar_FileSystem/results/sema3d/trainval_best):
0%| | 0/7 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/share/pycharm/helpers/pydev/pydevd.py", line 1599, in
globals = debugger.run(setup['file'], None, None, is_module)
File "/usr/share/pycharm/helpers/pydev/pydevd.py", line 1026, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/usr/share/pycharm/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/main.py", line 387, in
main()
File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/main.py", line 286, in main
acc, loss, oacc, avg_iou = train()
File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/main.py", line 169, in train
for bidx, (targets, GIs, clouds_data) in enumerate(loader):
File "/home/administrator/anaconda3/lib/python3.6/site-packages/tqdm/_tqdm.py", line 931, in iter
for obj in iterable:
File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 336, in next
return self._process_next_batch(batch)
File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 357, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
KeyError: 'Traceback (most recent call last):\n File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 106, in _worker_loop\n samples = collate_fn([dataset[i] for i in batch_indices])\n File "/home/administrator/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 106, in \n samples = collate_fn([dataset[i] for i in batch_indices])\n File "/home/administrator/anaconda3/lib/python3.6/site-packages/torchnet/dataset/listdataset.py", line 54, in getitem\n return self.load(self.list[idx])\n File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/spg.py", line 148, in loader\n cloud, diam = load_superpoint(args, db_path + '/superpoint_graphs/' + fname + '.h5', G.vs[s]['v'], train, test_seed_offset)\n File "/home/administrator/Documents.orig/Kumar_FileSystem/Server_Code/sematic3d/superpoint_graph-release/learning/spg.py", line 189, in load_superpoint\n P = hf['{:d}'.format(id)]\n File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper\n File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper\n File "/home/administrator/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 167, in getitem\n oid = h5o.open(self.id, self._e(name), lapl=self._lapl)\n File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper\n File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper\n File "h5py/h5o.pyx", line 190, in h5py.h5o.open\nKeyError: "Unable to open object (object '2324' doesn't exist)"\n'

Process finished with exit code 1

Semantic 3D for Training, Test reduced set custom

Hello,
I am trying to test some data I have, it is in x,y,z,i,i,i,i format to try and match the other reduced set .txts with x,y,z,i,r,g,b since I have no training data with labels so I must use Semantic3D for training.

Can I just train with the Semantic 3D data but test with custom .txts in Semantic3D format. I am unclear still from the documentation how I can achieve this? Do I have to retrain all over again to just test my data?

Tips on improving performance on custom dataset

Hi Loic,

I've been working on segmenting our own colorized point cloud data using your superpoint graph approach. Since annotating some of our own data will be time consuming (we will eventually do it), I first wanted to see how far I could get with the model pretrained on Semantic3D dataset. As of now, I'm able to get reasonable segmentation results, but there is still room for improvement on the accuracy. I just wanted to get your suggestions on whether any further improvement could be possible with quick fixes.

I went through all the relevant previous issues raised here on github to avoid wasting your time with tips you've already given others. Below is a list of changes I made on our data to make best possible use of Semantic3D model:

• Made sure the point clouds are sufficiently dense and that the density levels match Semantic3D data. Our scenes are much larger so I initially had to make them sparse enough for segmentation as a whole. Those results were not very good, so instead I've tiled the dense data and I'm processing it piece by piece. There are around 16 million points per input subset.

• Manually cleaned up outlier points below the ground level. Our data is obtained by a structure-from-motion algorithm on image sequences collected on a drone, so the nature of the outlier points are different from LIDAR collects. The performance was noticeably improved on the clean point clouds.

• Made sure that z axis points up in our dataset, that the units are in meters and building heights are not drastically different from those in Semantic3D.

• Tried offsetting the z values so that the ground level is around 0 and all z values in each point cloud are positive. This improved performance in certain locations, and made it worse in others (I have some examples for both).

• Made sure our data is in the same format as Semantic3D data, and the changes I made to the code mostly consists of fixing paths and filenames. I'm using the same features, same number of classes, same voxel size and reg strength, same colorization scheme etc.

• Looked at the _geof files to verify, to the best of my ability, that the color-coded geometric features made sense compared to those in Semantic3D data. Also verified that superpoint partitioning looks reasonable.

Most noticeable issues in the results we get:

• Incorrect labeling of roof points: The flat roofs of buildings are either labeled as man-made terrain or natural terrain. Is this because the Semantic3D training data does not feature flat roofs extensively? Some of the scenes have the roof segments visible for buildings, so I was expecting this to work fine.
• Trees are labeled as natural terrain with original z values, and as building with offset z values. Is this because the distribution of points around trees are different for SFM vs LIDAR?
• Vehicles labeled as hardscape with original z values, and partially labeled as building with offset z values.
• I am not sure if I understand the impact of height normalization that you talked about in some of the previous posts. As far as I'm aware, I did not change anything in that regard but I'm not sure if I need to.

Bug in Readme

When executing:

python partition/write_Semantic3D.py --SEMA3D_PATH $SEMA3D_DIR --odir "results/sema3d/best" --db_test_name testred

Error:
python: can't open file 'partition/write_Semantic3D.py': [Errno 2] No such file or directory

Bug: 'write_Semantic3D.py' should have a lowercase d
Best,
Max

Why is the cut-pursuit folder empty?

My friend, I want to compile libcp libraries, but there seems to be nothing in the cut-pursuit folder. Did I just miss something?

Issue with building libply_c

Hello,

I am following the step-by-step build instructions and I am running into a problem when trying to build libply_c. More specifically, when I run 'make', I get the following error message:

[ 50%] Building CXX object CMakeFiles/ply_c.dir/ply_c.cpp.o
~/partition/ply_c/ply_c.cpp:8:33: fatal error: numpy/ndarrayobject.h: No such file or directory
compilation terminated.
CMakeFiles/ply_c.dir/build.make:62: recipe for target 'CMakeFiles/ply_c.dir/ply_c.cpp.o' failed
make[2]: *** [CMakeFiles/ply_c.dir/ply_c.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/ply_c.dir/all' failed
make[1]: *** [CMakeFiles/ply_c.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

As far as I can tell, numpy is installed and ndarrayobject.h is present in two locations:

~/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h
~/anaconda3/pkgs/numpy-1.14.0-py36h3dfced4_1/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h

If I edit CMakeLists.txt file to add the following line right before INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_DIRS}), it seems to build fine with some warnings:

set(PYTHON_INCLUDE_DIRS ${PYTHON_INCLUDE_DIRS} ~/anaconda3/lib/python3.6/site-packages/numpy/core/include/)

I run into the same problem when building cut-pursuit as well. Is this edit to CMakeLists.txt file necessary? Or am I missing something else? I have multiple python versions installed, could that be a potential issue?

Thanks

How to Easily Test Data on Already Trained Data

I already trained my data on part of training and part of a withheld set. so I have a pth.tar file.

I am still confused as to what the terminal command I should use to write the labels for the test data I want to analyze.

I tried this command: python partition/write_Semantic3d.py --SEMA3D_PATH /media/amanda/Seagate\ Expansion\ Drive//Semantic3D_13 --odir "results/custom12/validation_best" --db_test_name testred

with my own custom classes (in same format as Semantic3D) and it complains that it doesn't have files:
(unable to open file: name = './results/custom12/validation_best/predictions_testred.h5')

What am I missing to just run test data on my trained NN? The documentation is a little unclear. Do I have to repartition, create superpoints, again for newly added test data?

Bug in provider.py:write_features

On line 341:
data_file.create_dataset('rgb', data=rgb, dtype='uint8')
On line 342:

    if len(rgb) > 0:
        data_file.create_dataset('rgb', data=rgb, dtype='uint8')

This causes problems while trying to save features_file.

Traceback (most recent call last):
  File "/home/sharma/miniconda3/envs/laser_int/lib/python3.6/pdb.py", line 1667, in main
    pdb._runscript(mainpyfile)
  File "/home/sharma/miniconda3/envs/laser_int/lib/python3.6/pdb.py", line 1548, in _runscript
    self.run(statement)
  File "/home/sharma/miniconda3/envs/laser_int/lib/python3.6/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/sharma/laser-integration/superpoint_graph/partition/partition.py", line 6, in <module>
    """
  File "partition/provider.py", line 343, in write_features
    data_file.create_dataset('rgb', data=rgb, dtype='uint8')
  File "/home/sharma/miniconda3/envs/laser_int/lib/python3.6/site-packages/h5py/_hl/group.py", line 109, in create_dataset
    self[name] = dset
  File "/home/sharma/miniconda3/envs/laser_int/lib/python3.6/site-packages/h5py/_hl/group.py", line 277, in __setitem__
    h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)

Problem of applying the SPG on other dataset!

Hi, thanks a lot for your great work. Firstly, I can sucessfully run the code on the semantic3D dataset, now I want to apply the SPG on my own dataset. I have converted the dataset to the format of the semantic3D, the specifical file can be showed as below. Here, my labels number of the dataset is 5, I can successfully run the partition.py and avoid the .h5 files, but when I run the main.py for training, it failured as the picture below:

How can I handle this problem?Could you please give me some suggestions? Thanks a lot for your kind help!
The data files and label files can be showed as below:

Questions about applying SPG to our own point cloud data

Hello,
Thank you very much for your great work! It really impressed us and we want to try your code on our own large scale outdoor point cloud data. Here I have some questions:

1. In which folder should I put your Sementic3D pre-trained model? Seems that it is mentioned in the readme.
2. Currently there is no RGB information in our data. We plan to just add 0 0 0 after each point's position to match Sementic3D data format. Is that OK?
3. If we want to directly test your pre-trained model on our new data, do we still have to download all the train data from Sementic3D and do the training for 1 epoch before testing? I suppose this is what happens with your current command line instruction.

Looking forward to your reply!

Question about the parameters in partition part

Hello Loic,

It's me again :D Currently we are trying to apply your network on our dataset，and we found that the segmentation result is not good. However, your partition part works pretty well and we plan to continue our work based on the partition result. So could you please tell me the exact meaning of the hyperparameters in partition part? I saw your help comments of these parameters, but some are a bit confusing to me.

Also, is it possible to only run the partition part and get the partition ply without running the network? Do we need a good GPU to run the partition part?

Looking forward to your reply!

Yongchi

Training problem

when i try CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset sema3d --SEMA3D_PATH $SEMA3D_DIR --db_test_name testred --db_train_name trainval
--epochs 500 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11
--nworkers 2 --odir "results/sema3d/trainval_best".and i was run it on Nvidia 1080Ti.
-> Train accuracy: 62.34092131206309, Loss: 1.289891004562378
Epoch 99/500 (results/sema3d/trainval_best):
0%| | 0/5 [00:00<?, ?it/s]Ptn with 850 clouds
20%|█████████████ | 1/5 [00:00<00:03, 1.08it/s]Ptn with 858 clouds
40%|██████████████████████████ | 2/5 [00:01<00:01, 1.51it/s]Ptn with 951 clouds
60%|███████████████████████████████████████ | 3/5 [00:01<00:01, 1.77it/s]Ptn with 677 clouds
80%|████████████████████████████████████████████████████ | 4/5 [00:01<00:00, 2.05it/s]Ptn with 680 clouds
100%|█████████████████████████████████████████████████████████████████| 5/5 [00:02<00:00, 2.26it/s]
0%| | 0/4 [00:00<?, ?it/s]Ptn with 3655 clouds
/home/dilu/Downloads/superpoint_graph-release/learning/pointnet.py:133: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
out = Variable(out.data, requires_grad=model.training, volatile=not model.training) # cut autograd
25%|████████████████▎ | 1/4 [00:02<00:07, 2.34s/it]Ptn with 2887 clouds
50%|████████████████████████████████▌ | 2/4 [00:02<00:02, 1.33s/it]Ptn with 14142 clouds
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "/home/dilu/Downloads/superpoint_graph-release/learning/main.py", line 387, in
main()
File "/home/dilu/Downloads/superpoint_graph-release/learning/main.py", line 290, in main
acc_test, oacc_test, avg_iou_test, avg_acc_test = eval()
File "/home/dilu/Downloads/superpoint_graph-release/learning/main.py", line 225, in eval
embeddings = ptnCloudEmbedder.run(model, *clouds_data)
File "/home/dilu/Downloads/superpoint_graph-release/learning/pointnet.py", line 132, in run_full_monger
out = model.ptn(Variable(clouds, volatile=True), Variable(clouds_global, volatile=True))
File "/home/dilu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/dilu/Downloads/superpoint_graph-release/learning/pointnet.py", line 94, in forward
input = self.convs(input)
File "/home/dilu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/dilu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/dilu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/dilu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 176, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58

Loading of trained network

I am trying to load the trained network as in instruction, but can't seem to unzip the tar file from the link.

I was running following command:

tar -xvzf model_sema3d_trainval.pth.tar
And receive the following error message:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Any idea why this is happening?

ImportError: No module named 'libcp'

I get the import error when I try to run:

python partition/write_Semantic3d.py --SEMA3D_PATH $SEMA3D_DIR --odir "results/sema3d/best" --db_test_name testred
Traceback (most recent call last):
  File "partition/write_Semantic3d.py", line 20, in <module>
    import libcp
ImportError: No module named 'libcp'

Partitioning based on geometric features only

Hi,

Running your cut-pursuit purely on geometric features of point cloud, without any RGB, produces the following output:

Graph 1571 vertices and 37656 edges and observation of dimension 4
Iteration 1 - 1 components - Saturation 100.0 % - Quadratic Energy nan % - Timer 0.025178
All components are saturated

I think this means the problem(graph) is ill conditioned so no iterations and thus, a degenerate solution? or maybe I am doing something silly.

Thanks for your reply.

The method mentioned in your paper

In your paper: WEAKLY SUPERVISED SEGMENTATION-AIDED CLASSIFICATION OF URBAN SCENES FROM3D LIDAR POINT CLOUDS
You mentioned that you implemented two other methods: pointwise and CRF regularization.
Could you share the code? So I can compare the two methods.
Thank you!

Strange value of the test accuracy

I run
CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset sema3d --SEMA3D_PATH $SEMA3D_DIR --db_test_name testred --db_train_name trainval
--epochs 500 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11
--nworkers 2 --odir "results/sema3d/trainval_best"
And I use epochs 100
After training, the result is
[{"avg_iou_test": 0.0, "acc_test": 1.1444885172178565, "epoch": 0, "oacc_test": 0.0, "per_class_iou_test": {"veget_hi": 0.0, "building": 0.0, "artefact": 0.0, "veget_low": 0.0, "scape": 0.0, "terrain_man": 0.0, "terrain_nature": 0.0, "cars": 0.0}, "avg_acc_test": 0.0}]
The test accuracy is very low and many metric is zeros.
I want to ask if I make some mistakes during training.
Thank you

Why the cut-pursuit folder is