m-3lab / real3d-ad

[NeurIPS 2023] Offical code for <Real3D-AD: A Dataset of Point Cloud Anomaly Detection>. A 3D point cloud anomaly detection dataset and benchmark.

Real3D-AD: A Dataset of Point Cloud Anomaly Detection

Jiaqi Liu*, Guoyang Xie*, Ruitao Chen*, Xinpeng Li, Jinbao Wang†, Yong Liu, Chengjie Wang, and Feng Zheng†

(* Equal contribution; † Corresponding authors)

Our paper has been accepted by NeurIPS 2023 Datasets & Benchmarks Track. [Paper]

Overview

This project aims to construct a new dataset of high-resolution 3D point clouds for anomaly detection tasks in real-world scenes.

Real3D-AD can be used for training and testing 3D anonmaly detection algorithms.

Note that different from RGB + Depth patterns, we only provide 3D point clouds for users.

Real3D-AD

Summary

• overview of all classes in Real3D-AD

Real3D-AD comprises a total of 1,254 samples that are distributed across 12 distinct categories. These categories include Airplane, Car, Candybar, Chicken, Diamond, Duck, Fish, Gemstone, Seahorse, Shell, Starfish, and Toffees.

Download

• To download the Real3D-AD dataset (Dataset for training and evaluation, pcd format), click real3d-ad-pcd.zip(google drive) or real3d-ad-pcd.zip(baidu disk: vrmi)
• To download the Real3D-AD dataset (Source data from camera, ply format), click real3d-ad-ply.zip(google drive) or real3d-ad-ply.zip(baidu disk：vvz1)

Data preparation

• Download real3d-ad-pcd.zip and extract into ./data/

data
├── airplane
    ├── train
        ├── 1_prototype.pcd
        ├── 2_prototype.pcd
        ...
    ├── test
        ├── 1_bulge.pcd
        ├── 2_sink.pcd
        ...
    ├── gt
        ├── 1_bulge.txt
        ├── 2_sink.txt
        ... 
├── car
...

Checkpoint preparation

Backbone	Pretrain Method
Point Transformer	Point-MAE
Point Transformer	Point-Bert

• Download checkpoints and move them into ./checkpoints/

Dataset Statistic

• brief describe our dataset in Table

Source	Class	Real Size [mm] (length/width/height)	Transparency	TrainingNum (good)	TestNum (good)	TestNum (defect)	TotalNum	TrainingPoints (min/max/mean)	TestPoints (min/max/mean)	AnomalyProportion Δ
1	Airplane	34.0/14.2/31.7	Yes	4	50	50	104	383k/ 413k/ 400k	168k/ 773k/351k	1.17%
2	Car	35.0/29.0/12.5	Yes	4	50	50	104	566k/1296k/1097k	90k/ 149k/131k	1.98%
3	Candybar	33.0/20.0/ 8.0	Yes	4	50	50	104	339k/1183k/ 553k	149k/ 180k/157k	2.36%
4	Chicken	25.0/14.0/20.0	No (white)	4	52	54	110	217k/1631k/1157k	87k/1645k/356k	4.46%
5	Diamond	29.0/29.0/18.7	Yes	4	50	50	104	1477k/2146k/1972k	66k/ 84k/ 75k	5.40%
6	Duck	30.0/22.2/29.4	Yes	4	50	50	104	545k/2675k/1750k	155k/ 784k/216k	1.99%
7	Fish	37.7/24.0/ 4.0	Yes	4	50	50	104	230k/ 251k/ 240k	104k/ 117k/110k	2.85%
8	Gemstone	22.5/18.8/17.0	Yes	4	50	50	104	169k/1819k/ 835k	43k/ 645k/104k	2.06%
9	Seahorse	38.0/11.2/ 3.5	Yes	4	50	50	104	189k/ 203k/ 194k	74k/ 90k/ 83k	4.57%
10	Shell	21.7/22.0/ 7.7	Yes	4	52	48	104	280k/ 316k/ 295k	110k/ 144k/125k	2.25%
11	Starfish	27.4/27.4/ 4.8	Yes	4	50	50	104	198k/ 209k/ 202k	74k/ 116k/ 88k	4.46%
12	Toffees	38.0/12.0/10.0	Yes	4	50	50	104	178k/1001k/ 385k	78k/ 97k/ 88k	2.46%

(Δ: Mean proportion of abnormal point clouds in Test set)

Data Collection

• description of instruments

The PMAX-S130 optical system comprises a pair of lenses with low distortion properties, a high luminance LED, and a blue-ray filter. The blue light scanner is equipped with a lens filter that selectively allows only the blue light of a specific wavelength to pass through. The filter effectively screens the majority of blue light due to its relatively low concentration in both natural and artificial lighting. Nevertheless, using blue light-emitting light sources could pose a unique obstacle in this context. The image sensor can collect light using the lens aperture. Hence, the influence exerted by ambient light is vastly reduced.

• how to capture point clouds and complete one prototype

Initially, the stationary object undergoes scanning while the turntable completes a full revolution of 360°, enabling the scanner to capture images of the various facets of the object. Subsequently, the object undergoes reversal, and the process of rotation and scanning is reiterated. Following the manual calibration of the front and back scanning outcomes, the algorithm performs a precise calibration of the stitching process. If there are any gaps in the stitching outcome, the scan stitching process is reiterated until the point cloud is rendered.

• anomalies

The anomalies pertaining to point clouds can be classified into two categories: incompleteness and redundancy. In the dataset, we named them bulge and sink. Besides, more samples are made by copying and cutting edges.

Annotation

• how to annotate

The collected point clouds are annotated using CloudCompare software CloudCompare is a 3D point cloud (grid) editing and processing software. Originally, it was designed to directly compare dense three-dimensional point clouds. It relies on a specific octree structure and provides excellent performance for tasks such as point cloud comparison. The anotation process of point cloud is shown in the figure below.

Benchmark

• beseline methods

We take BTF and M3DM as basic baseline methods, and improve baseline using PatchCore.

• metrics

We choose AUROC and AUPU as metric for object level and point level anomaly detection.

• benchmark results

Other methods on Real3D-AD

Complementary Pseudo Multimodal Feature for Point Cloud Anomaly Detection [paper 2023][code] Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network [paper 2023][code]

• PointCore: Efficient Unsupervised Point Cloud Anomaly Detector Using Local-Global Features [paper 2024]

	Object AUROC	Point AUROC	Max F1 Point	Point AP
airplane	0.632	0.618	0.023	0.010
candybar	0.518	0.836	0.135	0.064
car	0.718	0.734	0.107	0.050
chicken	0.640	0.559	0.071	0.031
diamond	0.640	0.753	0.149	0.074
duck	0.554	0.719	0.042	0.018
fish	0.840	0.988	0.582	0.559
gemstone	0.349	0.449	0.020	0.007
seahorse	0.843	0.962	0.615	0.636
shell	0.393	0.725	0.052	0.025
starfish	0.526	0.800	0.202	0.128
toffees	0.845	0.959	0.460	0.391
MEAN	0.625	0.758	0.205	0.166

Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network [paper][code]

The 3D anomaly detection performance when using 4 prototypes for training.

Object AUROC	BTF_FPFH	BTF_Raw	M3DM_PointMAE	M3DM_PointBERT	PatchCore+FPFH	PatchCore+FPFH+raw	PatchCore+PointMAE	Our baseline
airplane	0.730	0.520	0.434	0.407	0.882	0.848	0.726	0.716
car	0.647	0.560	0.541	0.506	0.590	0.777	0.498	0.697
candybar	0.703	0.462	0.450	0.442	0.565	0.626	0.585	0.827
chicken	0.789	0.432	0.683	0.673	0.837	0.853	0.827	0.852
diamond	0.707	0.545	0.602	0.627	0.574	0.784	0.783	0.900
duck	0.691	0.784	0.433	0.466	0.546	0.628	0.489	0.584
fish	0.602	0.549	0.540	0.556	0.675	0.837	0.630	0.915
gemstone	0.686	0.648	0.644	0.617	0.370	0.359	0.374	0.417
seahorse	0.596	0.779	0.495	0.494	0.505	0.767	0.539	0.762
shell	0.396	0.754	0.694	0.577	0.589	0.663	0.501	0.583
starfish	0.530	0.575	0.551	0.528	0.441	0.471	0.519	0.506
toffees	0.539	0.630	0.552	0.562	0.541	0.570	0.663	0.685
Average	0.635	0.603	0.552	0.538	0.593	0.682	0.594	0.704

Object AUPR	BTF_FPFH	BTF_Raw	M3DM_PointMAE	M3DM_PointBERT	PatchCore+FPFH	PatchCore+FPFH+raw	PatchCore+PointMAE	Our baseline
airplane	0.659	0.506	0.479	0.497	0.852	0.807	0.747	0.703
car	0.653	0.523	0.508	0.517	0.611	0.766	0.555	0.753
candybar	0.638	0.490	0.498	0.480	0.553	0.611	0.576	0.824
chicken	0.814	0.464	0.739	0.716	0.872	0.885	0.864	0.884
diamond	0.677	0.535	0.620	0.661	0.569	0.767	0.801	0.884
duck	0.620	0.760	0.533	0.569	0.506	0.560	0.488	0.588
fish	0.638	0.633	0.525	0.628	0.642	0.844	0.720	0.939
gemstone	0.603	0.598	0.663	0.628	0.411	0.411	0.444	0.454
seahorse	0.567	0.793	0.518	0.491	0.508	0.763	0.546	0.787
shell	0.434	0.751	0.616	0.638	0.573	0.553	0.590	0.646
starfish	0.557	0.579	0.573	0.573	0.491	0.473	0.561	0.491
toffees	0.505	0.700	0.593	0.569	0.506	0.559	0.708	0.721
Average	0.614	0.611	0.572	0.581	0.591	0.667	0.633	0.723

Point AUROC	BTF_FPFH	BTF_Raw	M3DM_PointMAE	M3DM_PointBERT	PatchCore+FPFH	PatchCore+FPFH+raw	PatchCore+PointMAE	Our baseline
airplane	0.738	0.564	0.530	0.523	0.471	0.556	0.579	0.631
car	0.708	0.647	0.607	0.593	0.643	0.740	0.610	0.718
candybar	0.864	0.735	0.683	0.682	0.637	0.749	0.635	0.724
chicken	0.693	0.608	0.735	0.790	0.618	0.558	0.683	0.676
diamond	0.882	0.563	0.618	0.594	0.760	0.854	0.776	0.835
duck	0.875	0.601	0.678	0.668	0.430	0.658	0.439	0.503
fish	0.709	0.514	0.600	0.589	0.464	0.781	0.714	0.826
gemstone	0.891	0.597	0.654	0.646	0.830	0.539	0.514	0.545
seahorse	0.512	0.520	0.561	0.574	0.544	0.808	0.660	0.817
shell	0.571	0.489	0.748	0.732	0.596	0.753	0.725	0.811
starfish	0.501	0.392	0.555	0.563	0.522	0.613	0.641	0.617
toffees	0.815	0.623	0.679	0.677	0.411	0.549	0.727	0.759
Average	0.730	0.571	0.637	0.636	0.577	0.680	0.642	0.705

Point AUPR	BTF_FPFH	BTF_Raw	M3DM_PointMAE	M3DM_PointBERT	PatchCore+FPFH	PatchCore+FPFH+raw	PatchCore+PointMAE	Our baseline
airplane	0.027	0.012	0.007	0.007	0.027	0.016	0.016	0.017
car	0.028	0.014	0.018	0.017	0.034	0.160	0.069	0.135
candybar	0.118	0.025	0.016	0.016	0.142	0.092	0.020	0.109
chicken	0.044	0.049	0.310	0.377	0.040	0.045	0.052	0.044
diamond	0.239	0.032	0.033	0.038	0.273	0.363	0.107	0.191
duck	0.068	0.020	0.011	0.011	0.055	0.034	0.008	0.010
fish	0.036	0.017	0.025	0.039	0.052	0.266	0.201	0.437
gemstone	0.075	0.014	0.018	0.017	0.093	0.066	0.008	0.016
seahorse	0.027	0.031	0.030	0.028	0.031	0.291	0.071	0.182
shell	0.018	0.011	0.022	0.021	0.031	0.049	0.043	0.065
starfish	0.034	0.017	0.040	0.040	0.037	0.035	0.046	0.039
toffees	0.055	0.016	0.021	0.018	0.040	0.055	0.055	0.067
Average	0.064	0.022	0.046	0.052	0.071	0.123	0.058	0.109

How to reproduce our benchmark

• environment preparation

We implement benchmark under CUDA 11.3 Our environment can be reproduced by the following command:

conda env create -f real3dad.yaml
# Note that point2_ops_lib may need to be installed by the following command：
pip install "git+git://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"

• how to train and evaluation

Note: Although we have fixed the random seed, slight variations in the results may still occur due to other random factors. We will try to address this issue in future work.

sh start.sh

The result will output four different metrics (Object/Point AUROC/AUPR).

• how to visualize abnormal map

In util.visualization.py, we provide the function ''vis_pointcloud_gt'' to visualize ground truth with our gt files. Also, with a saved anomaly map xx.npy(n values between 0 and 1) and the corresponding pcd file(n xyz points), you can use "vis_pointcloud_anomalymap(point_cloud, anomaly_map)" to visualize anomaly regions.

Acknowledgments.

This work is supported by the National Key R&D Program of China (Grant NO. 2022YFF1202903) and the National Natural Science Foundation of China (Grant NO. 62122035, 62206122).

Our benchmark is built on BTF and M3DM and PatchCore, thanks their extraordinary works!

Thanks to all the people who worked hard to capture the data, especially Xinyu Tang for his efforts.

License

The dataset is released under the CC BY 4.0 license.

BibTex Citation

If you find this paper and repository useful, please cite our paper☺️.

@inproceedings{liu2023real3d,
  title={Real3D-AD: A Dataset of Point Cloud Anomaly Detection},
  author={Liu, Jiaqi and Xie, Guoyang and Li, Xinpeng and Wang, Jinbao and Liu, Yong and Wang, Chengjie and Zheng, Feng and others},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}

real3d-ad's Issues

Is there a supplementary material for this paper, and how do I get it?

I remember that NeuroIPS Track on Datasets and Benchmarks requires a supplementary material, which I think can help me better understand this dataset.

关于benchmark测试脚本的报错

作者您好, 在debian11下配好环境之后我运行start.sh中在相关地方出现报错

在main_patchcore_fpfh_raw.py等代码中concatenate时候会报错

main_patchcore_fpfh_raw.py

Traceback (most recent call last):
  File "main_patchcore_fpfh_raw.py", line 226, in <module>
    main() 
    ............
  File "main_patchcore_fpfh_raw.py", line 180, in run
    ap_mask = np.concatenate(np.asarray(masks_gt))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (100,) + inhomogeneous part.

main_patchcore_pointmae.py

INFO:__main__:Computing evaluation metrics.
Traceback (most recent call last):
  File "main_patchcore_pointmae.py", line 210, in <module>
    main()
  .................
  File "main_patchcore_pointmae.py", line 156, in run
    ap_mask = np.concatenate(np.asarray(masks_gt))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (100,) + inhomogeneous part.

main_reg3dad.py

INFO:__main__:Computing evaluation metrics.
Traceback (most recent call last):
  File "main_reg3dad.py", line 236, in <module>
    main()
  ........
  File "main_reg3dad.py", line 180, in run
    ap_mask = np.concatenate(np.asarray(masks_gt))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (100,) + inhomogeneous part.

我修改的地方只有在第一次运行的时候找不到 pointmae_pretrain.pth, 然后我将下载下来的权重 pretrain.pth 改名成前者

What is the difference between the two dataset?

---

What is the difference between the two dataset shown in the picture above?

about setting environment

When I excute code:

conda env create -f real3dad.yaml

there are some errors.

Can you supply any general or stepbystep information again? Thanks.

The number of points used in the experiments

How many points are used in the experiments? The num_points in the code doesn't seem to be used.

点云文件名参数含义

请问一下，如图所示，文件名里面前缀数字以及部分文件cut是什么含义？

About faiss-gpu

When I set faiss_on_gpu to True in main_rgd3d.py, the code runs for a while and then encounters an error message as follows. It was run on a single 3090 GPU. I would like to ask if you set faiss_on_gpu to True during your experiments? Have you encountered such errors?

Faiss assertion 'err == cudaSuccess' failed in virtual void faiss::gpu::StandardGpuResourcesImpl::deallocMemory(int, void*) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:518; details: Failed to cudaFree pointer 0x7fd0a603da00 (error 3 initialization error)
Traceback (most recent call last):
File "main_reg3dad.py", line 243, in
main()
File "/home/l/anaconda3/envs/real3dad/lib/python3.7/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/l/anaconda3/envs/real3dad/lib/python3.7/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/l/anaconda3/envs/real3dad/lib/python3.7/site-packages/click/core.py", line 1689, in invoke
return _process_result(rv)
File "/home/l/anaconda3/envs/real3dad/lib/python3.7/site-packages/click/core.py", line 1626, in _process_result
value = ctx.invoke(self._result_callback, value, **ctx.params)
File "/home/l/anaconda3/envs/real3dad/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "main_reg3dad.py", line 136, in run
test_loader
File "/home/l/proj/Real3D-AD-main/patchcore/patchcore.py", line 560, in predict_pmae
return self._predict_dataloader_pmae(data)
File "/home/l/proj/Real3D-AD-main/patchcore/patchcore.py", line 574, in _predict_dataloader_pmae
_scores, _masks = self._predict_pmae(input_pointcloud)
File "/home/l/proj/Real3D-AD-main/patchcore/patchcore.py", line 582, in _predict_pmae
features, sample_dix = self._embed_pointmae(input_pointcloud)
File "/home/l/proj/Real3D-AD-main/patchcore/patchcore.py", line 156, in _embed_pointmae
reg_data = get_registration_np(point_cloud.squeeze(0).cpu().numpy(),self.basic_template)
File "/home/l/proj/Real3D-AD-main/feature_extractors/ransac_position.py", line 113, in get_registration_np
voxel_size)
File "/home/l/proj/Real3D-AD-main/feature_extractors/ransac_position.py", line 99, in execute_global_registration
], o3d.registration.RANSACConvergenceCriteria(100000, 1000))
File "/home/l/anaconda3/envs/real3dad/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4977) is killed by signal: Aborted.

有没有predict的代码或者管道

单个pca文件的推理，以及显示

关于异常的类型

你好！非常感谢你们提出了点云异常数据集。我有点问题想请教一下：1.我似乎并没有在数据集中看到文章中所说的incompleteness类型的异常，只有sink和bulge类型的异常？2.gt中带有后缀cut和不带后缀cut的文件的差异是什么呢？