dvlab-research / stratified-transformer

Stratified Transformer for 3D Point Cloud Segmentation (CVPR 2022)

Stratified Transformer for 3D Point Cloud Segmentation

Xin Lai^*, Jianhui Liu^*, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, Jiaya Jia

This is the official PyTorch implementation of our paper Stratified Transformer for 3D Point Cloud Segmentation that has been accepted to CVPR 2022. [arXiv] [CVF]

Highlight

1. Our method (Stratified Transformer) achieves the state-of-the-art performance on 3D point cloud semantic segmentation on both S3DIS and ScanNetv2 datasets. It is the first time for a point-based method to outperform the voxel-based ones, such as SparseConvNet and MinkowskiNet;
2. Stratified Transformer is point-based, and constructed by Transformer with standard multi-head self-attention, enjoying large receptive field, robust generalization ability as well as competitive performance;
3. This repository develops a memory-efficient implementation to combat the issue of variant-length tokens with several CUDA kernels, avoiding unnecessary momery occupation of vacant tokens. We also use shared memory for further acceleration.

Get Started

Environment

1. Install dependencies

pip install -r requirements.txt

If you have any problem with the above command, you can also install them by

pip install torch_sparse==0.6.12
pip install torch_points3d==1.3.0
pip install tensorboard timm termcolor tensorboardX

2. Compile pointops

Make sure you have installed gcc and cuda, and nvcc can work (Note that if you install cuda by conda, it won't provide nvcc and you should install cuda manually.). Then, compile and install pointops2 as follows. (We have tested on gcc==7.5.0 and cuda==10.1)

cd lib/pointops2
python3 setup.py install

Datasets Preparation

S3DIS

Please refer to https://github.com/yanx27/Pointnet_Pointnet2_pytorch for S3DIS preprocessing. Then modify the data_root entry in the .yaml configuration file.

ScanNetv2

Please refer to https://github.com/dvlab-research/PointGroup for the ScanNetv2 preprocessing. Then change the data_root entry in the .yaml configuration file accordingly.

Training

S3DIS

• Stratified Transformer

python3 train.py --config config/s3dis/s3dis_stratified_transformer.yaml

• 3DSwin Transformer (The vanilla version shown in our paper)

python3 train.py --config config/s3dis/s3dis_swin3d_transformer.yaml

ScanNetv2

• Stratified Transformer

python3 train.py --config config/scannetv2/scannetv2_stratified_transformer.yaml

• 3DSwin Transformer (The vanilla version shown in our paper)

python3 train.py --config config/scannetv2/scannetv2_swin3d_transformer.yaml

Note: It is normal to see the the results on S3DIS fluctuate between -0.5% and +0.5% mIoU maybe because the size of S3DIS is relatively small, while the results on ScanNetv2 are relatively stable.

Testing

For testing, first change the model_path, save_folder and data_root_val (if applicable) accordingly. Then, run the following command.

python3 test.py --config [YOUR_CONFIG_PATH]

Pre-trained Models

For your convenience, you can download the pre-trained models and training/testing logs from Here.

Citation

If you find this project useful, please consider citing:

@inproceedings{lai2022stratified,
  title={Stratified Transformer for 3D Point Cloud Segmentation},
  author={Lai, Xin and Liu, Jianhui and Jiang, Li and Wang, Liwei and Zhao, Hengshuang and Liu, Shu and Qi, Xiaojuan and Jia, Jiaya},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={8500--8509},
  year={2022}
}

stratified-transformer's Issues

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 - Colab

Hello!

i'm trying to test this amazing architecture on google colab and I've only used the commands on the github guide. I post them to be clear

!git clone https://github.com/dvlab-research/Stratified-Transformer.git
!pip install -r /content/Stratified-Transformer/requirements.txt
!python3 /content/Stratified-Transformer/lib/pointops2/setup.py install

The third returns the error in the title but it is raised by the following:
"x86_64-linux-gnu-gcc: error: src/pointops_api.cpp: No such file or directory"

To be clear at most I write all the output of the line 3:

running install
running bdist_egg
running egg_info
writing pointops2.egg-info/PKG-INFO
writing dependency_links to pointops2.egg-info/dependency_links.txt
writing top-level names to pointops2.egg-info/top_level.txt
/usr/local/lib/python3.7/dist-packages/torch/utils/cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'pointops2.egg-info/SOURCES.txt'
writing manifest file 'pointops2.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
/usr/local/lib/python3.7/dist-packages/torch/utils/cpp_extension.py:813: UserWarning: The detected CUDA version (11.1) has a minor version mismatch with the version that was used to compile PyTorch (11.3). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
building 'pointops2_cuda' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.7/dist-packages/torch/include -I/usr/local/lib/python3.7/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.7/dist-packages/torch/include/TH -I/usr/local/lib/python3.7/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.7m -c src/pointops_api.cpp -o build/temp.linux-x86_64-3.7/src/pointops_api.o -g -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1013" -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
x86_64-linux-gnu-gcc: error: src/pointops_api.cpp: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

How can I solve?

About the code realization of key sampling

Hi, I have read through the code of stratified transformer, but I do not understand how to realize the key sampling with nearby points densely sampled and distant points sparsely sampled. How to realize and where? Thanks.

Segmentation fault (core dumped)

Hi:

I have install the environment now, but I have met an error. I use one GPU in the .yaml file. I have met the following error.

Segmentation fault (core dumped)
Have you ever met this error?

Best.

Environment issue from torch_sparse

Hello!

I installed all libraries need for the environment, however, I am not able to run the script. I get the same error with torch_sparse. My gcc is gcc-8. I don't know whether it can be a reason.

Thank you! I hope this problem will be resolved soon.

RuntimeError: CUDA error: too many resources requested for launch

Thanks for your excellent work!
When I train the s3dis_stratified_transformer, I meet the following the error? The version of torch, cuda is 1.9.0 and 10.2 respectively. Have you ever met this ?

/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at /opt/conda/conda-bld/pytorch_1623448224956/work/aten/src/ATen/native/BinaryOps.cpp:467.)
return torch.floor_divide(self, other)
/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/functional.py:652: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /opt/conda/conda-bld/pytorch_1623448224956/work/c10/core/TensorImpl.h:1156.)
return torch.max_pool1d(input, kernel_size, stride, padding, dilation, ceil_mode)
/root/miniconda3/envs/point/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown
len(cache))
/root/miniconda3/envs/point/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 17 leaked semaphores to clean up at shutdown
len(cache))
/root/miniconda3/envs/point/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown
len(cache))
Traceback (most recent call last):
File "exp/s3dis/stratified_transformer/train.py", line 585, in
main()
File "exp/s3dis/stratified_transformer/train.py", line 80, in main
mp.spawn(main_worker, nprocs=args.ngpus_per_node, args=(args.ngpus_per_node, args))
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 2 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/root/Project/Stratify-TR/exp/s3dis/stratified_transformer/train.py", line 345, in main_worker
loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, criterion, optimizer, epoch, scaler, scheduler)
File "/root/Project/Stratify-TR/exp/s3dis/stratified_transformer/train.py", line 418, in train
output = model(feat, coord, offset, batch, neighbor_idx)
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/root/Project/Stratify-TR/model/stratified_transformer.py", line 556, in forward
feats, xyz, offset, feats_down, xyz_down, offset_down = layer(feats, xyz, offset)
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/root/Project/Stratify-TR/model/stratified_transformer.py", line 393, in forward
feats = blk(feats, xyz, index_0, index_1, index_0_offsets, n_max)
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/root/Project/Stratify-TR/model/stratified_transformer.py", line 295, in forward
feats = self.attn(feats, xyz, index_0, index_1, index_0_offsets, n_max) # index_0 MUST be in ascending order
File "/root/miniconda3/envs/point/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/root/Project/Stratify-TR/model/stratified_transformer.py", line 235, in forward
attn_flat = attn_flat + relative_position_bias #[M, num_heads]
RuntimeError: CUDA error: too many resources requested for launch
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

How to write test results as ply

Hi,
I was lucky to try your code and test it on the S3DIS dataset, also I noticed the presence of "write_ply_color" and "write_ply_rgb" but it is not obvious how to be used in the test code
so, if there a way to do so, it would be appreciated.
specifically, how can points argument be passed to the functions

computation complexity

I mentioned the code process the point cloud data by voxelize and sampling，it because the compute of transformer on point data is too large, but for test ,do you have better idea to predict all the raw point data at one time

code for visualize

Excuse me , may I ask you whether have the code for visualizatioin。 And the training time is so long , i want to ask which cost the most of the time, is the model training or data load cost too much time

The training settings are different from the released config files.

In Page 6 of the paper, it says that "For ScanNetv2, we train for 600 epochs with weight decay and batch size set to 0.1 and 8 respectively". But I find that in the released config files, the network is trained by 100 epochs with a based learning rate of 0.006. Should I follow the config file to reproduce the results?

RunTimeError: Caught an unknown exception

I passed the following inputs to BasicLayer:
feats.shape= [2,256,256] ,two batches each one has dimension 256*256 , I have reshaped them to size= [256,512]
xyz.shape=[2,3,256] reshaped to [3,512]
each batch has two scenes, each scene has 256 points with 256 features for each one size =[256,256] for each scene (point cloud)
I putted the offset =[256,512] which means that the first point cloud ends at 256, the second at 512.
I got the following error in this line:
pointops_cuda.attention_step1_forward_cuda_v2

what would be the source of this error and how it might be resolved?
thank you so much for your help ..

can you share your dataset/s3dis/list/s3dis_names.txt and dataset/s3dis/list/val5_full.txt?

about multi gpu processing

Hi, authors,

Thanks a lot for releasing the code. Does the current package only support multi-gpu processing (as it's found that SyncBN is used)? How could I run the training code if only a single GPU is allowed?

Thanks~

Running setup.py install for torch-points-kernels ... error

I got some error when I pip install torch_points3d==1.3.0 and pip install torch_points_kernels==0.6.10
how can I do ? Could you help me ? Thank you !

Failed to build torch-points-kernels
Installing collected packages: torch-points-kernels
Running setup.py install for torch-points-kernels ... error
ERROR: Command errored out with exit status 1:
command: /home/dq/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6p0cpdov/torch-points-kernels_42643aa2263b4566b10bdad314b648dc/setup.py'"'"'; file='"'"'/tmp/pip-install-6p0cpdov/torch-points-kernels_42643aa2263b4566b10bdad314b648dc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-qr4jjkpc/install-record.txt --single-version-externally-managed --compile --install-headers /home/dq/anaconda3/include/python3.8/torch-points-kernels
cwd: /tmp/pip-install-6p0cpdov/torch-points-kernels_42643aa2263b4566b10bdad314b648dc/
Complete output (22 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/chamfer_dist.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/knn.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/torchpoints.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/cluster.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/cubic_feature_sampling.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/init.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/gridding.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
copying torch_points_kernels/metrics.py -> build/lib.linux-x86_64-3.8/torch_points_kernels
running build_ext
building 'torch_points_kernels.points_cuda' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cuda
creating build/temp.linux-x86_64-3.8/cuda/src
gcc -pthread -B /home/dq/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icuda/include -I/home/dq/anaconda3/lib/python3.8/site-packages/torch/include -I/home/dq/anaconda3/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/dq/anaconda3/lib/python3.8/site-packages/torch/include/TH -I/home/dq/anaconda3/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.0/include -I/home/dq/anaconda3/include/python3.8 -c cuda/src/ball_query.cpp -o build/temp.linux-x86_64-3.8/cuda/src/ball_query.o -O3 -DVERSION_GE_1_3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=points_cuda -DTORCH_EXTENSION_NAME=points_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
gcc: error trying to exec 'cc1plus': execvp: 没有那个文件或目录
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/dq/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6p0cpdov/torch-points-kernels_42643aa2263b4566b10bdad314b648dc/setup.py'"'"'; file='"'"'/tmp/pip-install-6p0cpdov/torch-points-kernels_42643aa2263b4566b10bdad314b648dc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-qr4jjkpc/install-record.txt --single-version-externally-managed --compile --install-headers /home/dq/anaconda3/include/python3.8/torch-points-kernels Check the logs for full command output.

compiling pointnet2 error

could you please explain if I can run this code with cuda 10.0 pytorch 1.4 , my gpu driver does not support newer pytorch versions
I have compiled pointnet2 but I got this error :
error: command '/usr/bin/gcc' failed with exit code 1
my gcc version is 5.4.0
After compilation, I think this is the only module that are not created
No module named "pointops2_cuda"
is there any way to compile this lib without the need to upgrade my pytorch version?
Thanks in advance

在运行中出现TypeError: 'int' object is not iterable还有float不能索引等 TypeError,是否是环境问题？

s3dis数据集，修改了train_gpu: [0]

这个地方提示intersection_meter.sum是一个int类型外面加sum就会报错，TypeError: 'int' object is not iterable

这里iou_class[i]在索引时，会出现'float' object is not subscriptable，debug查看到这里是一个float数

是我的环境配置出了问题吗？
中间环境配置过程也确实好几个一直报错，用的是cuda 11.0，30系列显卡不支持cuda10.2，在配置过程中torch-scatter，torch-points-kernels安装试了很久，会出现AttributeError: ‘NoneType‘ object has no attribute ‘origin‘解决办法（安装geometric报错），参考的：https://blog.csdn.net/weixin_39417324/article/details/116159791 ；才终于安装上

是因为我的环境有问题吗？

Environmental problem

Thanks for all you have contributed to the open community!
I encountered a familiar bug with #36 when run the command python3 train.py --config config/s3dis/s3dis_stratified_transformer.yaml, which is detailed as follows:
`Traceback (most recent call last):
File "train.py", line 21, in
from util.s3dis import S3DIS
File "/root/Stratified-Transformer/util/s3dis.py", line 8, in
from util.voxelize import voxelize
File "/root/Stratified-Transformer/util/voxelize.py", line 4, in
from torch_geometric.nn import voxel_grid
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch_geometric/init.py", line 5, in
import torch_geometric.data
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch_geometric/data/init.py", line 1, in
from .data import Data
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch_geometric/data/data.py", line 8, in
from torch_sparse import coalesce, SparseTensor
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch_sparse/init.py", line 40, in
from .storage import SparseStorage # noqa
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch_sparse/storage.py", line 21, in
class SparseStorage(object):
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch/jit/_script.py", line 924, in script
_compile_and_register_class(obj, _rcb, qualified_name)
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch/jit/_script.py", line 64, in _compile_and_register_class
torch._C._jit_script_class_compile(qualified_name, ast, defaults, rcb)
RuntimeError:
Arguments for call are not valid.
The following variants are available:

aten::div.Tensor(Tensor self, Tensor other) -> (Tensor):
Expected a value of type 'Tensor' for argument 'other' but instead found type 'int'.

aten::div.Scalar(Tensor self, Scalar other) -> (Tensor):
Keyword argument rounding_mode unknown.

aten::div.out(Tensor self, Tensor other, *, Tensor(a!) out) -> (Tensor(a!)):
Expected a value of type 'Tensor' for argument 'other' but instead found type 'int'.

aten::div.int(int a, int b) -> (float):
Keyword argument rounding_mode unknown.

aten::div.float(float a, float b) -> (float):
Expected a value of type 'float' for argument 'b' but instead found type 'int'.

aten::div(Scalar a, Scalar b) -> (float):
Keyword argument rounding_mode unknown.

div(float a, Tensor b) -> (Tensor):
Expected a value of type 'Tensor' for argument 'b' but instead found type 'int'.

div(int a, Tensor b) -> (Tensor):
Expected a value of type 'Tensor' for argument 'b' but instead found type 'int'.

The original call is:
File "/opt/anaconda3/envs/st/lib/python3.7/site-packages/torch_sparse/storage.py", line 316
idx = self.sparse_size(1) * self.row() + self.col()

    row = torch.div(idx, num_cols, rounding_mode='floor')
          ~~~~~~~~~ <--- HERE
    col = idx % num_cols
    assert row.dtype == torch.long and col.dtype == torch.long`

I use the same enviroment with requirements.txt, which torch==1.7.1, gcc==7, torch_sparse==0.6.12, torch_points3d==1.3.0. what should I do to fix the problem?
Thanks again!

在Evaluation阶段显存占用会突然提高，然后OOM

[07/22 17:59:29 main-logger]: >>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>
Traceback (most recent call last):
File "/media/zhu/软件/python projet/Stratified-Transformer/train_zhu.py", line 569, in
main()
File "/media/zhu/软件/python projet/Stratified-Transformer/train_zhu.py", line 86, in main
main_worker(args.train_gpu, args.ngpus_per_node, args)
File "/media/zhu/软件/python projet/Stratified-Transformer/train_zhu.py", line 342, in main_worker
loss_val, mIoU_val, mAcc_val, allAcc_val = validate(val_loader, model, criterion)
File "/media/zhu/软件/python projet/Stratified-Transformer/train_zhu.py", line 517, in validate
output = model(feat, coord, offset, batch, neighbor_idx)
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 159, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/media/zhu/软件/python projet/Stratified-Transformer/model/stratified_transformer.py", line 438, in forward
feats = layer(feats, xyz, batch, neighbor_idx)
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/media/zhu/软件/python projet/Stratified-Transformer/model/stratified_transformer.py", line 357, in forward
feats = self.kpconv(xyz, xyz, neighbor_idx, feats)
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch_points3d/modules/KPConv/kernels.py", line 92, in forward
self.aggregation_mode,
File "/home/zhu/anaconda3/envs/pct/lib/python3.7/site-packages/torch_points3d/modules/KPConv/convolution_ops.py", line 63, in KPConv_ops
sq_distances = torch.sum(differences ** 2, dim=3)
RuntimeError: CUDA out of memory. Tried to allocate 11.81 GiB (GPU 0; 23.70 GiB total capacity; 13.35 GiB already allocated; 7.47 GiB free; 14.86 GiB reserved in total by PyTorch)

进程已结束,退出代码1

Q1: Start Evaluation会OOM
对数据进行了放缩，已改善
我是在的其他是数据集跑的，把参数改小后不会OOM但是效果很差

通过把max_num_neighbors、k 以及grid_size调大后效果好了不少，但是局部还有一些不太好

再调整这些参数，提升很小了

-------更详细的config在最后

能给给一些建议吗？

Q2:由于pheno4d的数据集是没有颜色的，但是改写dataloader中feat涉及很多其他地方
coord, feat, label = data[:, 0:3], np.zeros_like(data[:,0:3]), data[:, 3]
我就简单的赋值为0的，有方法可以取消feat吗？
config中DATA: fea_dim: 3 好像修改没用

下面是我的config文件
pheno4d.zip

这是我跑100epoch 的结果--第4列是训练用的label，第6列是pred的label
pred_data1.zip
pred_data2.zip

Low training efficiency and warning "batch_size shortened from 8 to 1, points from 640000 to 80000."

Hello!

Thanks for your great work. I tried to implement your code in my workstation.

I found that the max_batch_points and voxel_max were set to 140000 and 80000 in s3dis_stratified_transformer.yaml, respectively.

And batch_size was set to 8. In this case, the number of points in one batch can easily exceed 14w, say, the maximum is 8w*8=64w.

Maybe that's the reason why I often got this kind of warning,

"WARNING [main-logger]: batch_size shortened from 8 to 1, points from 640000 to 80000."

If I set the batch size to 1, the warning disappears, but less batch size brings more steps. Since I observed a low training speed per step, the overall training efficiency per epoch was poor with the whole S3DIS dataset. Do you have any suggestions to improve it?

I trained some other models like KPConv in my workstation. The training speed was higher and acceptable.

May I know whether the transformer model is relatively low in training efficiency? I am a layman in ML.

Cheers,

Eric.

Could you share your shapenetpart implement details?

e.g how do you use the shape class embedding and what's the models hyper param that you used

Does anyone come across the problems of gcc version

_OSError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/liy0r/anaconda3/envs/stformer/lib/python3.7/site-packages/torch_spline_conv/basis_cuda.so)

Which model is used for testing on S3DIS?

Hi, which model is used for testing on S3DIS dataset in your paper, the best model or the last model? Thanks.

RuntimeError: Caught an unknown exception!

Hello, is there any hint for the cause of the problem?

  File "../lib/pointops2/functions/pointops.py", line 158, in forward
    pointops_cuda.attention_step1_forward_cuda_v2(N_k, M, h, C, n_max, q, k, index0_offsets, index1, output)
RuntimeError: Caught an unknown exception!

Training stratified transformer on other datasets

Hi, I have a question.

If I train this stratified transformer on my own dataset, what parameters should I change to achieve relatively good performance?

Best regards,

Eric

Visualization Code

Could you share your visualization code please~
Thanks a lot.

version？

Please ask what the python version used by the author is

stratified_transformer.py, stratified calss , forward function inputs ?

forward function for class Stratified takes these variables as input:
features, xyz, offset and batch
could you please explain what are these inputs refer to ? I wanna apply this architecture to a different computer vision task with different dataset, I know what are the features and xyz but what is offset?

thank you so much in advance..

CUDA error: device-side assert triggered

Hi, authors,

Thanks a lot for your awesome work.

I met this error, have you ever met it?

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
output = module(*input, **kwargs)
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/mmvc/Congcong/Stratified-Transformer/model/stratified_transformer.py", line 438, in forward
feats = layer(feats, xyz, batch, neighbor_idx)
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/mmvc/Congcong/Stratified-Transformer/model/stratified_transformer.py", line 357, in forward
feats = self.kpconv(xyz, xyz, neighbor_idx, feats)
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch_points3d/modules/KPConv/kernels.py", line 83, in forward
new_feat = KPConv_ops(
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch_points3d/modules/KPConv/convolution_ops.py",line 95, in KPConv_ops
neighborhood_features = gather(features, neighbors_indices)
File "/home/mmvc/anaconda3/envs/pytorch19/lib/python3.8/site-packages/torch_points3d/core/common_modules/gathering.py", line 10, in gather
idx[idx == -1] = x.shape[0] - 1 # Shadow point
RuntimeError: CUDA error: device-side assert triggered

RuntimeError: Unkown device

Hi, thanks for your great project firstly. However, when I was running your project, I encountered such an error below. I set gpus and multiprocessing_distributed in config according to my computer correctly, but still got this error. Could you please help me with it? Many thanks ahead

Traceback (most recent call last):
File "train.py", line 542, in
main()
File "train.py", line 84, in main
main_worker(args.train_gpu, args.ngpus_per_node, args)
File "train.py", line 306, in main_worker
loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, criterion, optimizer, epoch, scaler, scheduler)
File "train.py", line 363, in train
neighbor_idx = tp.ball_query(radius, args.max_num_neighbors, coord, coord, mode="partial_dense", batch_x=batch, batch_y=batch)[0]
File "/home/user/anaconda3/envs/torch3d/lib/python3.8/site-packages/torch_points_kernels/torchpoints.py", line 210, in ball_query
return ball_query_partial_dense(radius, nsample, x, y, batch_x, batch_y, sort=sort)
File "/home/user/anaconda3/envs/torch3d/lib/python3.8/site-packages/torch_points_kernels/torchpoints.py", line 167, in ball_query_partial_dense
ind, dist = tpcpu.batch_ball_query(x, y, batch_x, batch_y, radius, nsample, mode=0, sorted=sort)
RuntimeError: Unknown device: -27. If you have recently updated the caffe2.proto file to add a new device type, did you forget to update the DeviceTypeName() function to reflect such recent changes?

Pretrained models.

Hi,

Thanks for your work and the code. Could you share the pre-trained models that produce the great results in your paper?

Best.

INT_MAX error

Hello,

I am encountering the following error, when using the model on larger pointclouds. I have reduced the model size and batch size, however I still get the following error.

index_0 = p2v_map.unsqueeze(-1).expand(-1, -1, k)[mask_mat] #[M, ]
RuntimeError: nonzero is not supported for tensors with more than INT_MAX elements,   file a support request

Do you know what causes this? I havent been able to find anything useful online.

About conda list

Hello! Thank you for your awesome work!
May I have a look at your conda list?

The size of the attn_flat is [M, heads] , what is M ?

In this part of code ,
attn_flat = attn_flat + relative_position_bias #[M, num_heads]

softmax_attn_flat = scatter_softmax(src=attn_flat, index=index_0, dim=0) #[M, num_heads]

could you please explain what is M here? thank you so much

CUDA Illegal Memory Access

I have switched to test_attention_op_step1.py to bypass n_max and adapt the model to larger pointclouds, however I am facing

line 187, in forward
    relative_position = xyz[index_0] - xyz[index_1]
RuntimeError: CUDA error: an illegal memory access was encountered

CUDA_LAUNCHBLOCKING Debugging doesnt show any usefull information. Do I also need to downgrade other functions or modify the code of v1?

What is the "offset" in class "Basic layer" file "Startified_transformer.py"

thank you for this great work
in class "Basic layer" features size =NC, xyz=N3, how about the offset?
thank you again 👍

ERROR：RuntimeError: Error compiling objects for extension

Error processing line 1 of /home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/matplotlib-3.5.2-py3.8-nspkg.pth:

Traceback (most recent call last):
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site.py", line 169, in addpackage
exec(line)
File "", line 1, in
File "", line 553, in module_from_spec
AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
running install
/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
writing pointops2.egg-info/PKG-INFO
writing dependency_links to pointops2.egg-info/dependency_links.txt
writing top-level names to pointops2.egg-info/top_level.txt
Error processing line 1 of /home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/matplotlib-3.5.2-py3.8-nspkg.pth:

Traceback (most recent call last):
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site.py", line 169, in addpackage
exec(line)
File "", line 1, in
File "", line 553, in module_from_spec
AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
reading manifest file 'pointops2.egg-info/SOURCES.txt'
writing manifest file 'pointops2.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'pointops2_cuda' extension
Error processing line 1 of /home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/matplotlib-3.5.2-py3.8-nspkg.pth:

Traceback (most recent call last):
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site.py", line 169, in addpackage
exec(line)
File "", line 1, in
File "", line 553, in module_from_spec
AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Emitting ninja build file /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Error processing line 1 of /home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/matplotlib-3.5.2-py3.8-nspkg.pth:

Traceback (most recent call last):
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site.py", line 169, in addpackage
exec(line)
File "", line 1, in
File "", line 553, in module_from_spec
AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
[1/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/knnquery/knnquery_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/knnquery/knnquery_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/knnquery/knnquery_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/knnquery/knnquery_cuda.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/knnquery/knnquery_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/knnquery/knnquery_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/knnquery/knnquery_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/knnquery/knnquery_cuda.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[2/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/grouping/grouping_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/grouping/grouping_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/grouping/grouping_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/grouping/grouping_cuda.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/grouping/grouping_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/grouping/grouping_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/grouping/grouping_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/grouping/grouping_cuda.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[3/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/aggregation/aggregation_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/aggregation/aggregation_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/aggregation/aggregation_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/aggregation/aggregation_cuda.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/aggregation/aggregation_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/aggregation/aggregation_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/aggregation/aggregation_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/aggregation/aggregation_cuda.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[4/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention_v2/attention_cuda_v2.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/attention_v2/attention_cuda_v2.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention_v2/attention_cuda_v2.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention_v2/attention_cuda_v2.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention_v2/attention_cuda_v2.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/attention_v2/attention_cuda_v2.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention_v2/attention_cuda_v2.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/attention_v2/attention_cuda_v2.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[5/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention/attention_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/attention/attention_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention/attention_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention/attention_cuda.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention/attention_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/attention/attention_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/attention/attention_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/attention/attention_cuda.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[6/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe/relative_pos_encoding_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/rpe/relative_pos_encoding_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe/relative_pos_encoding_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe/relative_pos_encoding_cuda.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe/relative_pos_encoding_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/rpe/relative_pos_encoding_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe/relative_pos_encoding_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/rpe/relative_pos_encoding_cuda.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[7/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/interpolation/interpolation_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/interpolation/interpolation_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/interpolation/interpolation_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/interpolation/interpolation_cuda.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/interpolation/interpolation_cuda.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/interpolation/interpolation_cuda.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/interpolation/interpolation_cuda.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/interpolation/interpolation_cuda.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
[8/12] c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe_v2/relative_pos_encoding_cuda_v2.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/rpe_v2/relative_pos_encoding_cuda_v2.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe_v2/relative_pos_encoding_cuda_v2.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe_v2/relative_pos_encoding_cuda_v2.o
c++ -MMD -MF /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe_v2/relative_pos_encoding_cuda_v2.o.d -pthread -B /home/yn/anaconda3/envs/pytorch-gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/TH -I/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/yn/anaconda3/envs/pytorch-gpu/include/python3.8 -c -c /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/rpe_v2/relative_pos_encoding_cuda_v2.cpp -o /home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/build/temp.linux-x86_64-3.8/src/rpe_v2/relative_pos_encoding_cuda_v2.o -g -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=pointops2_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/yn/project/Stratified-Transformer-main/Stratified-Transformer-main/lib/pointops2/src/rpe_v2/relative_pos_encoding_cuda_v2.cpp:2:10: fatal error: THC/THC.h: 没有那个文件或目录
2 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1740, in _run_ninja_build
subprocess.run(
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "setup.py", line 12, in
setup(
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
return run_commands(dist)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
dist.run_commands()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
self.run_command(cmd)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/install.py", line 74, in run
self.do_egg_install()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/install.py", line 123, in do_egg_install
self.run_command('bdist_egg')
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 165, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
self.run_command(cmdname)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
self.build()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 107, in build
self.run_command('build_ext')
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 741, in build_extensions
build_ext.build_extensions(self)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 202, in build_extension
_build_ext.build_extension(self, ext)
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 528, in build_extension
objects = self.compiler.compile(sources,
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 562, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1419, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/home/yn/anaconda3/envs/pytorch-gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1756, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

The stratified transformer with a shifted window

Thanks for the great work, it is very inspiring!

In the paper, you mention to apply the stratified transformer with a shifted window. But I cannot figure out the corresponding implementation in the released code. Could you please tell me where the shifted window is used in your code?

How to reproduce the result in the paper.

I run for the model from scratch and the best mIoU is: 0.701

Error running train.py s3dis: The deleter and context arguments are mutually exclusive.

I have some errors when I run train.py based on your guideline on s3dis dataset. I also keep configuration files and install libraries like requirements.txt. Because I use Cuda 11.3, I install torch 1.10.0+cu113 and torch_geometric 1.10.0+cu113. Can you help me with this issue?

[05/27 14:35:26 main-logger]: #Model parameters: 7069114
[05/27 14:35:28 main-logger]: augmentation all
[05/27 14:35:28 main-logger]: jitter_sigma: 0.005, jitter_clip: 0.02
Totally 204 samples in train set.
[05/27 14:35:28 main-logger]: train_data samples: '6120'
Totally 67 samples in val set.
[05/27 14:35:28 main-logger]: scheduler: MultiStep. scheduler_update: epoch. milestones: [60, 80], gamma: 0.1
[05/27 14:35:28 main-logger]: lr: [0.006, 0.0006000000000000001]
WARNING [05/27 14:35:30 main-logger]: batch_size shortened from 2 to 1, points from 160000 to 80000
WARNING [05/27 14:35:30 main-logger]: batch_size shortened from 2 to 1, points from 160000 to 80000
WARNING [05/27 14:35:30 main-logger]: batch_size shortened from 2 to 1, points from 143440 to 63440
WARNING [05/27 14:35:31 main-logger]: batch_size shortened from 2 to 1, points from 157885 to 77885
WARNING [05/27 14:35:31 main-logger]: batch_size shortened from 2 to 1, points from 159069 to 80000
WARNING [05/27 14:35:31 main-logger]: batch_size shortened from 2 to 1, points from 160000 to 80000
torch.Size([108012, 3])
0.0025000000000000005
WARNING [05/27 14:35:32 main-logger]: batch_size shortened from 2 to 1, points from 144512 to 80000
WARNING [05/27 14:35:32 main-logger]: batch_size shortened from 2 to 1, points from 140900 to 66523
WARNING [05/27 14:35:33 main-logger]: batch_size shortened from 2 to 1, points from 160000 to 80000
Traceback (most recent call last):
File "train.py", line 543, in
main()
File "train.py", line 84, in main
main_worker(args.train_gpu, args.ngpus_per_node, args)
File "train.py", line 306, in main_worker
loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, criterion, optimizer, epoch, scaler, scheduler)
File "train.py", line 363, in train
neighbor_idx = tp.ball_query(radius, args.max_num_neighbors, coord, coord, mode="partial_dense", batch_x=batch, batch_y=batch)[0]
File "/home/pknu/anaconda3/envs/testPt/lib/python3.7/site-packages/torch_points_kernels/torchpoints.py", line 210, in ball_query
return ball_query_partial_dense(radius, nsample, x, y, batch_x, batch_y, sort=sort)
File "/home/pknu/anaconda3/envs/testPt/lib/python3.7/site-packages/torch_points_kernels/torchpoints.py", line 167, in ball_query_partial_dense
ind, dist = tpcpu.batch_ball_query(x, y, batch_x, batch_y, radius, nsample, mode=0, sorted=sort)
ValueError: The deleter and context arguments are mutually exclusive.

Cross Attention with attention_step1_v2 using differently sized key and query sets

Hi!

Thanks for this amazing project! I'd like to use your memory-efficient imlementation of sparse attention in another project.
In my project, I'd like to perform cross-attention between a query set (N_query=100) and a key set (N_key=35000) which have different sizes.

I adapted this test file:
https://github.com/dvlab-research/Stratified-Transformer/blob/main/lib/pointops2/functions/test_attention_op_step1_v2.py
to allow different sizes for the query and key set. Note the change in the 2nd line of code in the snippet below.

For the first version pointops.attention_step1, I obtain no errors and can perform cross-attention.
However for the second (more efficient?) version pointops.attention_step1_v2, I obtain a CUDA runtime error. RuntimeError: CUDA error: an illegal memory access was encountered when I try to access attn_flat_v2, e.g., print(attn_flat_v2).

Do I use the function correctly?

Thanks a lot for your help and keep up the great work!

Best,
Jonas

    M = 8000
    N_query, N_key = 100, 35000  # HAVE A DIFFERENT TOKEN LENGTH FOR KEYS AND QUERIES
    C = 96
    h = 6
    query = torch.rand(N_query, h, C // h).cuda()
    key = torch.rand(N_key, h, C // h).cuda()

    index_0 = torch.rand(M)
    index_0[index_0 < 0] = 0
    index_0 = (index_0 * N_query).long().cuda()

    index_1 = torch.rand(M)
    index_1[index_1 < 0] = 0
    index_1 = (index_1 * N_key).long().cuda()

    query.requires_grad = True
    key.requires_grad = True

    # rearrange index for acceleration
    index_0, indices = torch.sort(index_0)  # [M,]
    index_1 = index_1[indices]  # [M,]
    index_0_counts = index_0.bincount()
    n_max = index_0_counts.max()
    index_0_offsets = index_0_counts.cumsum(dim=-1)  # [N]
    index_0_offsets = torch.cat([torch.zeros(1, dtype=torch.long).cuda(), index_0_offsets], 0)  # [N+1]

    attn_flat = pointops.attention_step1(query.float(), key.float(), index_0.int(), index_1.int())
    attn_flat_v2 = pointops.attention_step1_v2(query.float(), key.float(), index_1.int(), index_0_offsets.int(),
                                               n_max)

Can anyone run the code successfully

Dear all,
Does someone successfully install the environment? Can you share with me your version of each third-party library? It looks like there are some libs conflicts when I installed following by the readme files

Will you update the shapenet part dataset config?

some question about 'n_max'

Hi @X-Lai ,
Thanks for the released code.

I am just wondering that why the 'n_max' need to be less than 1024, as you coded in class AttentionStep1_v2(Function). When I note the code 'assert n_max <= 1024', there will be an CUDA error: invalid configuration argument. (I think it is due to the limitation for 'n_max')

Sorry that I'm not familiar with CUDA C++ coding.
So, is there some solution to break the limitation for 'n_max'? Could you give me some advice?

It could be very nice that you leave me a email address, which could be convenient for discussion.
Here is mine: [email protected]

Wish every success in your work.

A environment bug

I find a bug when I implement from torch_geometric import voxel_grid.

Loss Nan

Hi, I'm reproducing your excellent work, the loss is reached to the nan with obtained 0 mIou. Is the learning rate set incorrectly？

aug: True
base_lr: 0.006
batch_size: 8
batch_size_test: 4
batch_size_val: 4
channels: [48, 96, 192, 384]
classes: 13
concat_xyz: True
data_name: s3dis
data_root: ../../Dataset/S3DIS/trainval_fullarea
depths: [2, 2, 6, 2]
dist_backend: nccl
dist_url: tcp://127.0.0.1:58501
distributed: True
downsample_scale: 8
drop_path_rate: 0.3
drop_rate: 0.5
epochs: 50
eval_freq: 1
evaluate: True
fea_dim: 6
grid_size: 0.04
grid_sizes: [0.04, 0.08, 0.16, 0.32]
ignore_label: 255
jitter_clip: 0.02
jitter_sigma: 0.005
k: 16
loop: 30
manual_seed: 123
max_batch_points: 140000
max_num_neighbors: 34
model_path: None
momentum: 0.9
multiplier: 0.1
multiprocessing_distributed: True
names_path: data/s3dis/s3dis_names.txt
ngpus_per_node: 4
num_heads: [3, 6, 12, 24]
num_layers: 4
optimizer: AdamW
patch_size: 0.04
print_freq: 1
quant_size: 0.01
quant_sizes: [0.01, 0.02, 0.04, 0.08]
rank: 0
ratio: 0.25
rel_key: True
rel_query: True
rel_value: True
resume: exp/s3dis/stratified_transformer/model/model_last.pth
save_folder: None
save_freq: 1
save_path: exp/s3dis/stratified_transformer/model
scheduler: MultiStep
scheduler_update: epoch
split: val
start_epoch: 0
stem_transformer: True
step_epoch: 30
sync_bn: True
test_area: 5
test_gpu: [0]
test_list: data/s3dis/list/val5.txt
test_list_full: data/s3dis/list/val5_full.txt
test_workers: 4
train_gpu: [0, 1, 2, 3]
transformer_lr_scale: 0.1
up_k: 3
use_amp: True
use_xyz: True
voxel_max: 80000
voxel_size: 0.04
warmup: linear
warmup_iters: 1500
warmup_ratio: 1e-06
weight: None
weight_decay: 0.01
window_size: [0.16, 0.32, 0.64, 1.28]
workers: 4
world_size: 4

�[32m[06/02 00:16:07 main-logger]: �[0mTrain result at epoch [23/50]: mIoU/mAcc/allAcc 0.0130/0.0769/0.1693.
WARNING:root:NaN or Inf found in input tensor.
�[32m[06/02 00:16:07 main-logger]: �[0m>>>>>>>>>>>>>>>> Start Evaluation >>>>>>>>>>>>>>>>
�[32m[06/02 00:16:28 main-logger]: �[0mTest: [1/17] Data 4.412 (4.412) Batch 21.110 (21.110) Loss nan (nan) Accuracy 0.2793.
�[32m[06/02 00:16:31 main-logger]: �[0mTest: [2/17] Data 0.002 (2.207) Batch 3.324 (12.217) Loss nan (nan) Accuracy 0.1809.
�[32m[06/02 00:16:36 main-logger]: �[0mTest: [3/17] Data 0.001 (1.472) Batch 4.249 (9.561) Loss nan (nan) Accuracy 0.1969.
�[32m[06/02 00:16:38 main-logger]: �[0mTest: [4/17] Data 0.001 (1.104) Batch 2.364 (7.762) Loss nan (nan) Accuracy 0.1815.
�[32m[06/02 00:16:48 main-logger]: �[0mTest: [5/17] Data 0.001 (0.883) Batch 10.526 (8.314) Loss nan (nan) Accuracy 0.1848.
�[32m[06/02 00:17:02 main-logger]: �[0mTest: [6/17] Data 0.001 (0.736) Batch 13.099 (9.112) Loss nan (nan) Accuracy 0.1507.
�[32m[06/02 00:17:08 main-logger]: �[0mTest: [7/17] Data 0.001 (0.631) Batch 5.935 (8.658) Loss nan (nan) Accuracy 0.1931.
�[32m[06/02 00:17:09 main-logger]: �[0mTest: [8/17] Data 0.001 (0.553) Batch 1.564 (7.771) Loss nan (nan) Accuracy 0.1636.
�[32m[06/02 00:17:10 main-logger]: �[0mTest: [9/17] Data 0.001 (0.491) Batch 1.386 (7.062) Loss nan (nan) Accuracy 0.1597.
�[32m[06/02 00:17:13 main-logger]: �[0mTest: [10/17] Data 0.000 (0.442) Batch 2.654 (6.621) Loss nan (nan) Accuracy 0.1734.
�[32m[06/02 00:17:15 main-logger]: �[0mTest: [11/17] Data 0.001 (0.402) Batch 2.049 (6.205) Loss nan (nan) Accuracy 0.2362.
�[32m[06/02 00:17:26 main-logger]: �[0mTest: [12/17] Data 0.001 (0.369) Batch 11.297 (6.630) Loss nan (nan) Accuracy 0.1980.
�[32m[06/02 00:17:29 main-logger]: �[0mTest: [13/17] Data 0.002 (0.340) Batch 2.617 (6.321) Loss nan (nan) Accuracy 0.1651.
�[32m[06/02 00:17:55 main-logger]: �[0mTest: [14/17] Data 0.001 (0.316) Batch 25.620 (7.700) Loss nan (nan) Accuracy 0.1804.
�[32m[06/02 00:18:04 main-logger]: �[0mTest: [15/17] Data 0.001 (0.295) Batch 9.624 (7.828) Loss nan (nan) Accuracy 0.1890.
�[32m[06/02 00:18:06 main-logger]: �[0mTest: [16/17] Data 0.001 (0.277) Batch 1.992 (7.463) Loss nan (nan) Accuracy 0.1893.
�[32m[06/02 00:18:09 main-logger]: �[0mTest: [17/17] Data 0.001 (0.260) Batch 2.720 (7.184) Loss nan (nan) Accuracy 0.1934.
�[32m[06/02 00:18:10 main-logger]: �[0mVal result: mIoU/mAcc/allAcc 0.0146/0.0769/0.1903.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_0 Result: iou/accuracy 0.1903/1.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_1 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_2 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_3 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_4 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_5 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_6 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_7 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_8 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_9 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_10 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_11 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0mClass_12 Result: iou/accuracy 0.0000/0.0000.
�[32m[06/02 00:18:10 main-logger]: �[0m<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<
WARNING:root:NaN or Inf found in input tensor.
�[32m[06/02 00:18:10 main-logger]: �[0mSaving checkpoint to: exp/s3dis/stratified_transformer/model/model_last.pth
�[32m[06/02 00:18:10 main-logger]: �[0mlr: [0.006, 0.0006000000000000001]
batch_size shortened from 2 to 1, points from 156591 to 80000
�[32m[06/02 00:18:20 main-logger]: �[0mEpoch: [24/50][1/765] Data 4.474 (4.474) Batch 10.469 (10.469) Remain 60:03:51 Loss nan Lr: [0.006, 0.0006] Accuracy 0.1705.

I really want to ask you a question about the function of attention

I saw a function called attention_step2_forward_cuda in the python file of pointops, but I couldn't find the definition of this function. I wrote a simple program to call the function, and the terminal displayed: RuntimeError: Caught an unknown exception! So I especially want to know The definition or function of this function!
Thanks and wish you all the best!

Training stops in epoch 10 with Assertion Error

Hey,

So I was training Stratified Transformer on my own dataset. Till epoch 10 everything worked fine and nothing was going wrong. But during epoch 10 this error is thrown below. It is related to the relative_position_index here in the position encoding part, where it checks if all the indices are positive.
I am just thinking why would any index be negative in the first place. A little more information, I am using global coordinates of points and am also doing some custom augmentations which flip and rotate the point cloud. During the first 10 epochs, the validation results are better than with simple augs, so the augs are working well.

As a solution, I thought if I made sure that the relative position here is always positive by taking absolute values of this subtraction, how would that affect the training.

Let me know what you think

Where is the paper

Hi, thanks for your code repo, but the paper 'Stratified Transformer for 3D Point Cloud Segmentation' was not found on google, maybe it is not public yet now, could you please attach a PDF of that?

pointops2 error

Hello, I'm sorry to bother you. I have a question. I use cuda10 2,cudnn7. 6.5, pytrch1. 11. I made an error in compiling pointops2. Have you ever encountered this problem and any solutions? I look forward to your answer. thank you

ERF visualization

Hi,

Thanks for your awesome work and the code.
Could you share the code for visualizing ERF as Fig 1 in your paper?

Best.

Environment Version

I have a problem with the installation of pyg.
Here is my environment: python=3.7, pytorch=1.8.0, cudatoolkit=11.1, other packages are default version.

Console returns:
''
Package libstdcxx-ng conflicts for:
pytorch==1.8.0 -> cudatoolkit[version='>=11.1,<11.2'] -> libstdcxx-ng[version='>=10.3.0|>=7.5.0|>=7.3.0|>=7.2.0|>=11.2.0']
gmp -> libstdcxx-ng[version='>=7.2.0|>=7.3.0|>=7.5.0']
libtiff -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=7.5.0|>=7.2.0']
torchvision==0.9.0 -> cudatoolkit[version='>=11.1,<11.2'] -> libstdcxx-ng[version='>=10.3.0|>=7.3.0|>=7.5.0|>=11.2.0|>=8.4.0|>=7.2.0']

....

• pytorch==1.8.0 -> cudatoolkit[version='>=11.1,<11.2'] -> __glibc[version='>=2.17,<3.0.a0']
• readline -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
• scipy -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
• sqlite -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
• tk -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
• torchvision==0.9.0 -> cudatoolkit[version='>=11.1,<11.2'] -> __glibc[version='>=2.17,<3.0.a0']
• xz -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
• zlib -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31
''
Can you list the specific versions of your environments? Thanks very much.