haochengwan / pvt

PVT: Point-Voxel Transformer for 3D Deep Learning

PVT: Point-Voxel Transformer for 3D Deep Learning

We will release the latest code of our PVTv4 at arXiv upon acceptance!

Paper and Citation

The paper can be downloaded from arXiv.
If you like our work and think it helpful to your project, please cite it as follows.

@article{zhang2021point,
  title={PVT: Point-Voxel Transformer for 3D Deep Learning},
  author={Zhang, Cheng and Wan, Haocheng and Liu, Shengqiang and Shen, Xinyi and Wu, Zizhao},
  journal={arXiv preprint arXiv:2108.06076},
  year={2021}
}

Prerequisites

The code is built with following libraries (see requirements.txt):

• Python >= 3.7
• PyTorch >= 1.3
• numba
• numpy
• scipy
• six
• tensorboardX >= 1.2
• plyfile
• h5py
• timm

Data Preparation

ModelNet40

Download alignment ModelNet40 here and save in data/modelnet40_normal_resampled/.

S3DIS

We follow the data pre-processing in PointCNN. The code for preprocessing the S3DIS dataset is located in data/s3dis/. One should first download the dataset from here, then run

python data/s3dis/prepare_data.py

ShapeNet

We follow the data pre-processing in PointNet2. Please download the dataset from hereand save in data/shapenetcore_partanno_segmentation_benchmark_v0_normal/.

KITTI

For Frustum-PointNet backbone, we follow the data pre-processing in Frustum-Pointnets. One should first download the ground truth labels from here, then run

unzip data_object_label_2.zip
mv training/label_2 data/kitti/ground_truth
./data/kitti/frustum/download.sh

Pretrained Models

Here we provide a pretrained model on ModelNet40. The accuracy might vary a little bit compared to the paper, since we re-train some of the models for reproducibility. The path of the model is in ./checkpoints/cls/model.t7

Example training and testing

#train
python main_cls.py --exp_name=cls --num_points=1024 --use_sgd=True --batch_size 32 --epochs 200 --lr 0.001

#test
python main_cls.py --exp_name=cls --num_points=1024 --use_sgd=True --eval=True --model_path=checkpoints/cls/model.t7 --test_batch_size 32

pvt's Issues

About the performance

Thanks for your owesome work!

Thanks for your opened code, it's very helpful for me!

I reproduced the released code on the S3DIS dataset, but I only got 0.552240 test iou in the training process, which is a certain gap from any version (v1-v4) of the paper. What is the reason? Here is my training log:

Namespace(batch_size=16, data_root='/home/lishuai375/data/PVT/s3dis', dataset='S3DIS', dropout=0.5, epochs=100, eval=False, exp_name='s3dis_semseg_baseline', lr=0.001, model='pvt', model_root='checkpoint/semseg', momentum=0.9, no_cuda=False, num_points=4096, scheduler='cos', seed=3407, test_area='5', test_batch_size=16, use_sgd=True)
test area: 5, loss: 2.207159, test acc: 0.472239, test avg acc: 0.347459, test iou: 0.241691
test area: 5, loss: 1.578993, test acc: 0.733327, test avg acc: 0.525323, test iou: 0.396903
test area: 5, loss: 1.513533, test acc: 0.739214, test avg acc: 0.585753, test iou: 0.425839
test area: 5, loss: 1.495074, test acc: 0.760526, test avg acc: 0.590498, test iou: 0.451339
test area: 5, loss: 1.417374, test acc: 0.804674, test avg acc: 0.574718, test iou: 0.464658
test area: 5, loss: 1.418835, test acc: 0.794469, test avg acc: 0.566810, test iou: 0.462191
test area: 5, loss: 1.453407, test acc: 0.787238, test avg acc: 0.578352, test iou: 0.457491
test area: 5, loss: 1.384028, test acc: 0.820696, test avg acc: 0.589148, test iou: 0.493558
test area: 5, loss: 1.353592, test acc: 0.823281, test avg acc: 0.602668, test iou: 0.507241
test area: 5, loss: 1.442432, test acc: 0.785170, test avg acc: 0.599437, test iou: 0.478820
test area: 5, loss: 1.384134, test acc: 0.813472, test avg acc: 0.591278, test iou: 0.498485
test area: 5, loss: 1.376553, test acc: 0.813226, test avg acc: 0.597232, test iou: 0.486455
test area: 5, loss: 1.417028, test acc: 0.801129, test avg acc: 0.555291, test iou: 0.467786
test area: 5, loss: 1.322940, test acc: 0.836944, test avg acc: 0.648626, test iou: 0.524083
test area: 5, loss: 1.379500, test acc: 0.814751, test avg acc: 0.582442, test iou: 0.489711
test area: 5, loss: 1.363020, test acc: 0.821304, test avg acc: 0.566748, test iou: 0.482473
test area: 5, loss: 1.343999, test acc: 0.823860, test avg acc: 0.617797, test iou: 0.512286
test area: 5, loss: 1.330696, test acc: 0.834802, test avg acc: 0.590888, test iou: 0.507075
test area: 5, loss: 1.377645, test acc: 0.821068, test avg acc: 0.575637, test iou: 0.494437
test area: 5, loss: 1.335137, test acc: 0.834010, test avg acc: 0.614604, test iou: 0.510045
test area: 5, loss: 1.370512, test acc: 0.814682, test avg acc: 0.629383, test iou: 0.511584
test area: 5, loss: 1.392636, test acc: 0.813674, test avg acc: 0.575049, test iou: 0.496428
test area: 5, loss: 1.384423, test acc: 0.813870, test avg acc: 0.593170, test iou: 0.491461
test area: 5, loss: 1.336348, test acc: 0.833097, test avg acc: 0.595234, test iou: 0.511293
test area: 5, loss: 1.363717, test acc: 0.821999, test avg acc: 0.575544, test iou: 0.486584
test area: 5, loss: 1.427415, test acc: 0.783737, test avg acc: 0.581958, test iou: 0.460246
test area: 5, loss: 1.342196, test acc: 0.831708, test avg acc: 0.622571, test iou: 0.523159
test area: 5, loss: 1.334389, test acc: 0.834026, test avg acc: 0.611450, test iou: 0.514988
test area: 5, loss: 1.339650, test acc: 0.829793, test avg acc: 0.648654, test iou: 0.529194
test area: 5, loss: 1.324503, test acc: 0.838046, test avg acc: 0.634818, test iou: 0.533004
test area: 5, loss: 1.325878, test acc: 0.835349, test avg acc: 0.624838, test iou: 0.527651
test area: 5, loss: 1.358474, test acc: 0.826425, test avg acc: 0.574235, test iou: 0.489496
test area: 5, loss: 1.358899, test acc: 0.825027, test avg acc: 0.576418, test iou: 0.483577
test area: 5, loss: 1.372272, test acc: 0.820532, test avg acc: 0.594015, test iou: 0.485528
test area: 5, loss: 1.334299, test acc: 0.835287, test avg acc: 0.633590, test iou: 0.512516
test area: 5, loss: 1.358278, test acc: 0.828517, test avg acc: 0.571670, test iou: 0.492384
test area: 5, loss: 1.343390, test acc: 0.830378, test avg acc: 0.609656, test iou: 0.507609
test area: 5, loss: 1.308632, test acc: 0.842174, test avg acc: 0.651405, test iou: 0.542564
test area: 5, loss: 1.342251, test acc: 0.828315, test avg acc: 0.622899, test iou: 0.520599
test area: 5, loss: 1.336127, test acc: 0.833776, test avg acc: 0.598722, test iou: 0.504399
test area: 5, loss: 1.332503, test acc: 0.835423, test avg acc: 0.619560, test iou: 0.523039
test area: 5, loss: 1.357433, test acc: 0.820996, test avg acc: 0.590024, test iou: 0.496783
test area: 5, loss: 1.315869, test acc: 0.845794, test avg acc: 0.630980, test iou: 0.532167
test area: 5, loss: 1.408260, test acc: 0.803767, test avg acc: 0.621404, test iou: 0.466000
test area: 5, loss: 1.314720, test acc: 0.843913, test avg acc: 0.631912, test iou: 0.534767
test area: 5, loss: 1.319884, test acc: 0.838763, test avg acc: 0.605425, test iou: 0.523677
test area: 5, loss: 1.329233, test acc: 0.837477, test avg acc: 0.632895, test iou: 0.527570
test area: 5, loss: 1.315601, test acc: 0.846239, test avg acc: 0.613458, test iou: 0.518729
test area: 5, loss: 1.334415, test acc: 0.835855, test avg acc: 0.620678, test iou: 0.521479
test area: 5, loss: 1.332208, test acc: 0.838914, test avg acc: 0.606027, test iou: 0.518186
test area: 5, loss: 1.352821, test acc: 0.831446, test avg acc: 0.583080, test iou: 0.499663
test area: 5, loss: 1.340809, test acc: 0.836515, test avg acc: 0.596245, test iou: 0.512768
test area: 5, loss: 1.337996, test acc: 0.837838, test avg acc: 0.596923, test iou: 0.514673
test area: 5, loss: 1.316531, test acc: 0.843550, test avg acc: 0.612390, test iou: 0.524677
test area: 5, loss: 1.323478, test acc: 0.843650, test avg acc: 0.603496, test iou: 0.530073
test area: 5, loss: 1.335624, test acc: 0.836328, test avg acc: 0.571085, test iou: 0.498427
test area: 5, loss: 1.320783, test acc: 0.841788, test avg acc: 0.634043, test iou: 0.531701
test area: 5, loss: 1.331172, test acc: 0.842131, test avg acc: 0.589806, test iou: 0.510885
test area: 5, loss: 1.309691, test acc: 0.848171, test avg acc: 0.619787, test iou: 0.533212
test area: 5, loss: 1.322135, test acc: 0.841479, test avg acc: 0.635021, test iou: 0.535108
test area: 5, loss: 1.338577, test acc: 0.840515, test avg acc: 0.616291, test iou: 0.525319
test area: 5, loss: 1.331651, test acc: 0.842013, test avg acc: 0.597964, test iou: 0.510480
test area: 5, loss: 1.315937, test acc: 0.844733, test avg acc: 0.639809, test iou: 0.544864
test area: 5, loss: 1.312542, test acc: 0.847906, test avg acc: 0.635747, test iou: 0.544174
test area: 5, loss: 1.346074, test acc: 0.833783, test avg acc: 0.579576, test iou: 0.492760
test area: 5, loss: 1.321430, test acc: 0.846906, test avg acc: 0.600660, test iou: 0.527392
test area: 5, loss: 1.325952, test acc: 0.845499, test avg acc: 0.597792, test iou: 0.517230
test area: 5, loss: 1.304355, test acc: 0.851450, test avg acc: 0.650892, test iou: 0.552240
test area: 5, loss: 1.338303, test acc: 0.837126, test avg acc: 0.576707, test iou: 0.498103
test area: 5, loss: 1.315864, test acc: 0.850527, test avg acc: 0.622858, test iou: 0.534810
test area: 5, loss: 1.313099, test acc: 0.849597, test avg acc: 0.614718, test iou: 0.535642
test area: 5, loss: 1.319322, test acc: 0.844790, test avg acc: 0.632804, test iou: 0.534552
test area: 5, loss: 1.315454, test acc: 0.848608, test avg acc: 0.635184, test iou: 0.542930
test area: 5, loss: 1.317009, test acc: 0.850092, test avg acc: 0.625485, test iou: 0.539138
test area: 5, loss: 1.320978, test acc: 0.849201, test avg acc: 0.591676, test iou: 0.519443
test area: 5, loss: 1.388863, test acc: 0.817274, test avg acc: 0.527224, test iou: 0.454946
test area: 5, loss: 1.308460, test acc: 0.853383, test avg acc: 0.624306, test iou: 0.541771
test area: 5, loss: 1.314958, test acc: 0.852565, test avg acc: 0.613766, test iou: 0.535713
test area: 5, loss: 1.324447, test acc: 0.846032, test avg acc: 0.621593, test iou: 0.539622
test area: 5, loss: 1.322947, test acc: 0.849832, test avg acc: 0.617503, test iou: 0.537317
test area: 5, loss: 1.346119, test acc: 0.837689, test avg acc: 0.616287, test iou: 0.523273
test area: 5, loss: 1.338811, test acc: 0.845812, test avg acc: 0.590953, test iou: 0.512003
test area: 5, loss: 1.318881, test acc: 0.845784, test avg acc: 0.613445, test iou: 0.527249
test area: 5, loss: 1.320358, test acc: 0.851123, test avg acc: 0.612816, test iou: 0.537741
test area: 5, loss: 1.316418, test acc: 0.853035, test avg acc: 0.620381, test iou: 0.540914
test area: 5, loss: 1.309855, test acc: 0.856322, test avg acc: 0.626275, test iou: 0.544827
test area: 5, loss: 1.313699, test acc: 0.855087, test avg acc: 0.616010, test iou: 0.536479
test area: 5, loss: 1.309410, test acc: 0.854497, test avg acc: 0.629250, test iou: 0.547557
test area: 5, loss: 1.326637, test acc: 0.849793, test avg acc: 0.614308, test iou: 0.534644
test area: 5, loss: 1.319056, test acc: 0.852779, test avg acc: 0.610441, test iou: 0.536871
test area: 5, loss: 1.315993, test acc: 0.854967, test avg acc: 0.608906, test iou: 0.535047
test area: 5, loss: 1.316201, test acc: 0.854774, test avg acc: 0.613718, test iou: 0.540718
test area: 5, loss: 1.323511, test acc: 0.853655, test avg acc: 0.612365, test iou: 0.537222
test area: 5, loss: 1.326175, test acc: 0.852812, test avg acc: 0.605059, test iou: 0.532117
test area: 5, loss: 1.319622, test acc: 0.854838, test avg acc: 0.610070, test iou: 0.534993
test area: 5, loss: 1.324867, test acc: 0.852087, test avg acc: 0.611691, test iou: 0.533088
test area: 5, loss: 1.324838, test acc: 0.852700, test avg acc: 0.609207, test iou: 0.534654
test area: 5, loss: 1.326967, test acc: 0.852281, test avg acc: 0.610141, test iou: 0.535405
test area: 5, loss: 1.329787, test acc: 0.851539, test avg acc: 0.606156, test iou: 0.531444
test area: 5, loss: 1.325097, test acc: 0.852542, test avg acc: 0.607056, test iou: 0.531011
best test iou: 0.552240

Looking forward to your reply！

got a KeyError, need help plz

Hi, I use test command python main_cls.py --exp_name=cls --num_points=1024 --use_sgd=True --eval=True --model_path=checkpoints/cls/model.t7 --test_batch_size 32, but got a KeyError.Is the problem caused by different dataset files? need your help plz.
你好，我运行测试命令python main_cls.py --exp_name=cls --num_points=1024 --use_sgd=True --eval=True --model_path=checkpoints/cls/model.t7 --test_batch_size 32，但遇到了 KeyError，是数据集文件不同导致的问题吗？需要您的帮助。
Error :

...
File ".../PVT-master/data.py", line 71, in _get_item
    cls = self.classes[self.datapath[index][0]]
KeyError: 'data/modelnet40_normal_resampled/plydata'

content in modelnet40_test.txt:

data/modelnet40_normal_resampled/ply_data_test0.h5
data/modelnet40_normal_resampled/ply_data_test1.h5

voxel branch

Is the current code using the SWA?Or still using the dense one?

GPU-based rule book code location

Where is the implementation of gpu-based rule book in the code?

Relative Attention in Point branch?

Hi, it's really a nice work, but I have some problems about point branch.

As described in paper, the Relative Attention is performed in the point branch to obtains significant improvements over not using this RPR term without adding extra training parameters. But it seems that there is not relative code in repo?

https://github.com/HaochengWan/PVT/blob/master/modules/pvtconv.py#L298

forward() missing 1 required positional argument: 'rel_pos'

When I wanted to train dataset, I met following problems.

What is "rel_pos"?

Perhaps shared_transformer.py file should add "rel_pos" parameter in Class SharedTransformer ?

About voxel branch

Thanks for your great work! @HaochengWan

Please tell me where the SWA implementation of voxel branch? I can't find it!

Looking forward your rely!

Can't run the code

Hi @zhangcheng828 ,when I run main_cls.py and I can get nothing . I have found that there is something wrong when import some function,such from model.pvt import pvt and so on .But I have no idea how to solve it .Have you ever encounted the same mistake ?

您好，我想问一下您的模型在ModelNet40分类时采用了几个编码器？

There are some bugs

Hi, it's a great work. But I find some bugs in it. rel_pos should be fed to point_features in PartPVTConv and SegPVTConv, in pvtconv.py

there are also bugs in shared_transformer.py SA_Layer use rel_pos,but Shared_Transformer don't receive it and don't feed it to SA_Layer

the code above are screenshot in github code, as follow:

Do you have the data loader on modelnet 40