laughtervv / sgpn Goto Github PK

Hi, @laughtervv , @Rotch ,
I wonder if you could offer the code for converting instance segmentation results into 3D bounding boxes of object detection, since I don't know how to compute the orientation of the bounding box.
Thanks a lot!

@laughtervv, i follow you instruction to run the S3DIS data through train.py, but i got the log that Batch: 9, loss: 3238.821436, grouperr: 0.348972, same: 0.000233, diff: nan, pos: 0.959794. i want to know if it is right that diff is nan and why. Thank you a lot, i am looking for you reply!

Hello,

I have successfully run your code on my own data, however I think something is going wrong. First, I formatted my point cloud and ground truth using the same structure as the Stanfordindoor3d dataset. Next, I created an .h5 file from this data. I then pointed to the relevant text files created for the dataset in your valid.py script and everything appears to be running.

However, it is taking a very long time to run (2 days). I have a GTX1080ti and 32GB of system RAM. Is this to be expected? It is not a very large mesh of a room that I am passing in (around 80 blocks). I have a feeling that I am running your script wrong; based on your paper, inference should take less than 2 seconds.

Thank you
Matthew

Hi!
Does anyone know what is "gap = 5e-3 " in the test file. Why this value? How to calculate or find this value. I am trying to run SGPN with my own dataset and this value has an influence on the output.

Thank you in advance

Very cool work! Could you kindly provide the model trained on ScanNet?

Hi @laughtervv ,

I've been trying to run your code on the cityscape dataset. However I've got the log as follows:
Batch: 19, loss: 7658.622900, grouperr: 0.190640, same: 1.000000, diff: 1.000000, pos: 0.000000
I don't know why ,could you please give me some advice?

Thanks a lot !

Hi, @laughtervv ,

I got the following error when I run valid.py

(py2.7_tf1.4) root@milton-ThinkCentre-M93p:/data/code6/SGPN# python valid.py 
Traceback (most recent call last):
  File "valid.py", line 6, in <module>
    from utils.test_utils import *
  File "/data/code6/SGPN/utils/test_utils.py", line 310, in <module>
    magma_cmap = mpl.cm.get_cmap('magma')
AttributeError: 'module' object has no attribute 'cm'

I can run python train.py successfully before running valid.py.
Could you suggest me how to fix it?

THX!

To evaluate test result, The Scannet Evaluation Page not found,
Could you please release your test.py and predict.py to me ?
Thanks a lot ! @laughtervv

Hello, I read the code on calculating the loss and I have some questions.

1. line 167:
   group_mat_label = tf.matmul(pts_group_label,tf.transpose(pts_group_label, perm=[0, 2, 1]))
   I do not understand the meaning of this operation. Could you give an simple example for it.

2. line 179~184:
   diffgroup_samesem_mat_label = tf.multiply(diffgroup_mat_label, samesem_mat_label)
   diffgroup_diffsem_mat_label = tf.multiply(diffgroup_mat_label, diffsem_mat_label)

   num_samegroup = tf.reduce_sum(samegroup_mat_label)
   num_diffgroup_samesem = tf.reduce_sum(diffgroup_samesem_mat_label)
   num_diffgroup_diffsem = tf.reduce_sum(diffgroup_diffsem_mat_label)
   I am not clear about these code.

3. line 191:
   pos = tf.multiply(samegroup_mat_label, pred_simmat)
   I know that it want to compute the distance in the same group. But I do not know why this sentence can work. Could you provide an simple example or some equation?

4. line 197:
   group_mask_weight = tf.matmul(group_mask, tf.transpose(group_mask, perm=[0, 2, 1]))
   I think that maybe this is the same opeartion as line 167, but I cannot undestand

5. line 209~210:
   Pr_obj = tf.reduce_sum(pts_semseg_label,axis=2)
   Pr_obj = tf.cast(Pr_obj, tf.float32)
   I guess that all element in Pr_obj may be 1. So what the function of it as a weight?

Hi vv:
i looking a solution to make the .h5 file,for training my own dataset,but i dont't know how to do, Could you give me a help?

I got this error when using my own pointnet2_semantic checkpoint -

(0) Not found: Key Fsem/biases not found in checkpoint [[node save/RestoreV2 (defined at train.py:133) ]]

Has anyone any clue how to solve it? This repository seems to be built off pointnet2 but I now get the feeling that a pointnet checkpoint may be required, that or a checkpoint from training with the part_seg script is required.

Hi, laughtervv
I have run you method on scannet, and I use the default set to generate point block, but the output point cloud have been changed. It's shape have been changed. What can I do?
Thanks.

@weiyuewang @laughtervv thanks for open sourcing the wonderfull work , i had few queries
Q1 have you trained the architecture on the available other dataset like semanttic Kitti and 3D dataset
Q2 If not trained can we follow the same training pipeline , if trained can you please share the pre-trained model
Q3 can we use the currently pre-trained model to test on custom dataset which less number of point cloud density

Thanks in advance

The official h5 file of ShapeNet Dataset does not have the seglabel key. I wonder how to prepare the proper ShapeNet Dataset for SGPN? Thx.

Hi, @laughtervv
I trained as what you suggested, first pretrain the segmentation model, then use only SIM loss for the first 5 epoch and finally use the total loss. The total loss dropped to 20 and stays after 100 epochs. However, when I do the prediction, the results are bad: mAP : 0.004, other data is also far away from what you gives in your paper. I wonder what happens. The loss of yours can be as low as how much?

Thanks.

Hi, @laughtervv ，I download the dataset from https://drive.google.com/file/d/1UjcXB2wMlLt5qwYPk5iSAnlhttl1AO9u/view?ts=5aecefd1

But I get the training loss is very large. this is my training log.

the loss value is about 20927.877148, any problem which I met?

Hi, laughervv,
Can you provide the meta folder file for us >_<

Hi, @laughtervv , @Rotch
Thanks for sharing you nice work. However, I wonder how to obtain the instance label for each point? Since S3DIS dataset only has semantic labels for each point ( 13 categories ) .
Besides, if you have already obtain the instance label for each point, what is the need for block merging?

Thanks a lot!

It would be appreciate if you could provide the dataset used in your code.
Thanks.

Hello, @laughtervv @weiyuewang . In the experiments of ShapeNet, I want to perform the DBSCAN clustering algorithm to generate instance labels. But the parameters of DBSCAN are hard to chooose.
Could you please provide the values of eps and min_samples of DBSCAN?
And I would be grateful if you could provide the instance labels of ShapeNet.
Thank you!

Hi,
First of all thanks for sharing your nice work!
I was trying to run test.py on the example data provided here which tries to use some Restore Directory:

checkpoint/stanford_ins_seg_groupmask11_fromgroup_recipweight_nopow2_lr4

It looks there to find the files:

pergroup_thres.txt

mingroupsize.txt

Now I was wondering if these files could also be provided, or alternatively what they contain.

Any help is much appreciated!

Hello People, I'm wondering how the authors of the paper tested their network on the ShapeNet dataset? Any ideas?

when i use the provided pre-trained model, train on area 1-6 except 5 and test on area 5, i get the admirable result and mAP is about 0.6.

but when i train on area 2-6 and test on area 1, mAP is about 0.4 and then test on area 2, mAP is about 0.1.

i try many versions of mingroupsize.txt and pergroup_thres.txt, but 0.1 is still the best result.

so i have the question about the pre-trained model, maybe i shoule retrain the pre-trained model in each cross-validation?

Thanks for your response.

ps:
after training 200 epochs，loss is about 300-400, and grouperr: 0.02 same: 0.01 diff: 0.001 pos: 0.004

Hi @laughtervv ,

I've been trying to run your pretrained S3DIS model on point clouds from the matterport dataset. However I've found that group merging is very slow, for instance on a room divided into about 80 blocks, around 10000 points per block, group merging takes 5 mins + per block (if running on one cpu). Maybe I'm doing something wrong, did you also encounter this?

A separate problem is that to load one room into memory can take 30+ GB RAM, though I guess you could store and load on demand by block the model's output?

Sorry for all the questions, but also I've found group instance segmentation results to be a bit shoddy, perhaps it's something to do with the mingroupsize.txt and the pergroup_thres.txt and other thresholds? How did you get these values? I'm currently using the ones you posted on Issue #8.

Thanks for any help.

hello, @laughtervv I would be grateful if you could describe how to prepare h5 file with nyu dataset. I read the paper many times,but I didn't get the steps in details.

Hi,
Thanks for sharing such a good work. I have one question about the code of model. In pointnet,we know there are two transform matrixs , which named T-net ,and in your code ,I can not find them. I check it in pointnet work ,and it is same to you ,could you tell me why ? thanks a lot

Hi @laughtervv
I downloaded the h5 data from
https://drive.google.com/uc?id=1UjcXB2wMlLt5qwYPk5iSAnlhttl1AO9u&export=download
and tried training the model but it gives following error -
data = f['data'][:]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/group.py", line 262, in getitem
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'data' doesn't exist)"

I assume it's related to the h5 files. Do you have any idea why this is happening?

Hi @laughtervv ,

Thanks for sharing your code,
I've been trying to figure out how to preprocess new room pointcloud data to run on your pretrained stanford dataset model, without much success.

Do you have any programs or pipelines for this? How did you convert the stanford point cloud txt file data into the 9 vector format? Along with the conversion into blocks?
There are some functions in utils.pc_util and utils.plyfile for loading ply files, but these also seem insufficient?

Thanks again.

Hello @laughtervv :
i've trained SGPN on our own dataset.but i don't know how to joint pre-trained pointnet model, when i directly put the pointnet model to restore_model path, some error occurred, like key Fsem/bias not found in checkpoint and so on. but in original pointnet model, Fsem variable does not exist. or poinetnet doesn't need to pre-train?
could you please help me about this problem.
Thanks.

I want to train shapenet on SGPN but I do not know how to feed the data? Anyone can help me pls?

Hello,

The README.md points to this GDrive link to fetch the pretrained model, but it seems to have only the following files. Is this correct?

checkpoint                          
epoch_200.ckpt.index
epoch_200.ckpt.data-00000-of-00001  
epoch_200.ckpt.meta

If so, can I kindly ask if it is possible to load it in tensorflow (plug-and-play) and how to do it?
e.g.

model = tf.keras.models.load_model('trained_models/epoch_200.ckpt.data-00000-of-00001')

yields

>OSError: SavedModel file does not exist at: trained_models/epoch_200.ckpt.data-00000-of-00001/{saved_model.pbtxt|saved_model.pb}

Thank you, Gaston.

Hello, I want to know which areas you use in your pre-trained model. So I know which area should I use when validing and testing.

Thank you!

In the commit from May 31 2019 by @weiyuewang, sha bb2280b, at file generate_results.py, line 149: incomplete "try" block - no exept or finally statement.

too much nan
Processsing: Shape [1] Block[66] nan [ 4.80224806 nan 10.58879836 nan nan nan nan nan 16.78238869 nan 8.25528313 nan 12.3036682

Hello， @laughtervv
I have downloaded the indoor3d_sem_seg_hdf5_data，but I have a error：

Fail to load modelfile: checkpoint/stanford_ins_seg/trained_models/
Traceback (most recent call last):
File "train.py", line 277, in
train()
File "train.py", line 177, in train
cur_data, cur_group, _, cur_seg = provider.loadDataFile_with_groupseglabel_stanfordindoor(cur_train_filename)
File "/data/zyu/wangyu/wangy/SGPN-master/provider.py", line 207, in loadDataFile_with_groupseglabel_stanfordindoor
group = f['pid'][:].astype(np.int32)#NxG
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/zyu/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 177, in getitem
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'pid' doesn't exist)"

I think it’s related to the train_hdf5_file_list.txt, but I don't how to write train_hdf5_file_list.txt and didn't get the steps in details. Can you give some suggestion?
Thanks for your response.

File "/home/saira/Downloads/SGPN-master/provider.py", line 110, in getDataFiles
return [line.rstrip() for line in open(list_filename)]
FileNotFoundError: [Errno 2] No such file or directory: '/media/hdd2/data/pointnet/stanfordindoor/valid_hdf5_file_list.txt'

Hello all,
I am not sure how I choose the three normalised location values for each point. In my available dataset I have only the xyz and rgb values.
How do I compute the missing values?

Many thanks in advance!

Hi @laughtervv
I will run train.py with h5 files and pre-trained model you gave.
I saw you set epochs = 200 in you code(train.py). I think it is too large to cost much time. So I must set epochs = ? to get an equal result to your result in your paper?

Hi @laughtervv !
Can you please share the scripts used for the baseline Seg-Cluster that you compare against?

Hi, thanks for your code! But when I run generate_result.py, the functionGroupMerging_old() missed. I found a similar function called GroupMerging in test_util.py, I use it instead. But in function BlockMergin(), a bug happens as follow:

File "/SGPN/utils/test_utils.py", line 182, in BlockMerging()
if volume[xx,yy,zz]!=-1 and volume_seg[xx,yy,zz]==groupseg[grouplabel[i]]:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

May you help me solve it? thanks a lot!!

By the way, when i run valid.py, the first number of each line is nan. Does it mean no instance detected?

When I run valid.py The following error appears:
Traceback (most recent call last):
File "valid.py", line 160, in
predict()
File "valid.py", line 139, in predict
ths, ths_, cnt = Get_Ths(pts_corr_val, seg, ins, ths, ths_, cnt)
File "/home/hpz8g4/hanrui/SGPN/utils/test_utils.py", line 57, in Get_Ths
hist, bin = np.histogram(pt[pts_in_curins_ind], bins=20)
File "<array_function internals>", line 6, in histogram
File "/home/hpz8g4/anaconda3/envs/hanrui/lib/python3.7/site-packages/numpy/lib/histograms.py", line 856, in histogram
decrement = tmp_a < bin_edges[indices]
IndexError: index -9223372036854775808 is out of bounds for axis 0 with size 21

Thank you for your help

Hi @laughtervv Thanks for your awesome work. I have confusion on group merging. The group id of some points with low confidence or small cardinality is assigned to -1. After then, you decreased the group id by 1 again in your code(line 151, test_util.py). Can you explain why?

un, cnt = np.unique(groupid, return_counts=True)
groupidnew = groupid.copy()
for ig, g in enumerate(un):
    if g == -1:
        continue
    **groupidnew[groupid==g] = (ig-1)**
    groupseg[(ig-1)] = groupseg.pop(g)
groupid = groupidnew

Hi @laughtervv
Thank you for open sourcing your great work!
I downloaded your pre-trained model from here (the link in README) and your h5 data from here (the link in issue#3) and used pergroup_thres.txt and mingroupsize.txt in issue#8.
However, I ran test.py using them and got mAP 0.47, this is lower than the mAP reported in your paper (mAP 0.5435).
Did I make some mistakes?
Here is the result I got. Thank you.

Instance Segmentation AP: [ 0.47166667  0.53958333  0.37652931  0.67391304  0.43280553  0.7375
  0.67703333  0.42900336  0.34550953  0.2         0.38823529  0.51387205
  0.32641463]
Instance Segmentation mAP: 0.470158929369
Semantic Segmentation IoU: [ 0.85866874  0.92186948  0.86302137  0.84426813  0.72262609  0.8738558
  0.89429596  0.82030826  0.79822212  0.72040915  0.7932732   0.74137266
  0.81986088]
Semantic Segmentation Acc: %f 0.7979087289103282