uav4geo / openpointclass Goto Github PK
View Code? Open in Web Editor NEWFast and memory efficient semantic segmentation of 3D point clouds. Runs on Windows, Mac and Linux.
License: GNU Affero General Public License v3.0
Fast and memory efficient semantic segmentation of 3D point clouds. Runs on Windows, Mac and Linux.
License: GNU Affero General Public License v3.0
We should add support for running training/inference on non-colored inputs.
Could improve classification quality at the expense of slower runtime.
We are working to train a lot of point cloud previosly well classified by Leica Cyclone 3DR. They have a different classificiation such curbs (74), sidewalk, (73), walls (100) amongs other. The request is... if the code can train more thant the ASPRS 1.4 23 classifications. We are happy to use the starndar ones and train my self classifications. The speed of the code its amazing. The other comments its that Potree dont support more classifications too. :(( but one problem at time.
Here some very well trained set. We are workig to share more. My idea its create different scenarios such, city, road, farm, town among other that may be can ne integrated in WebODM Platform to choose one of them when process the photogrametric data.
passenger_terminal_05_r0_10_s20.zip
marina_05_r2.zip
espigon_05_r2.zip
Input files: https://hub.dronedb.app/r/hedo88/err-opc
Command: ./pcclassify ../point_cloud.ply out.ply ../model.bin
Loading ../model.bin
Reading 1060056 points
Starting resolution: 0.2
Init scale 0 at 0.2 ...
Init scale 1 at 0.2 ...
Init scale 3 at 0.8 ...
Init scale 5 at 3.2 ...
Init scale 4 at 1.6 ...
Init scale 2 at 0.4 ...
Building scale 1 (160105 points) ...
Building scale 2 (37145 points) ...
Building scale 3 (10084 points) ...
Building scale 4 (2694 points) ...
Building scale 5 (731 points) ...
Features: 105
Classifying...
Local smoothing...
Segmentation fault
with valgrind
:
Classifying...
Local smoothing...
==656== Thread 3:
==656== Invalid write of size 1
==656== at 0x17F432: rf::classify(PointSet&, liblearning::RandomForest::RandomForest<liblearning::RandomForest::NodeGini<liblearning::RandomForest::AxisAlignedSplitter> >*, std::vector<Feature*, std::allocator<Feature*> > const&, std::vector<Label, std::allocator<Label> > const&, rf::Regularization, bool, bool) [clone ._omp_fn.3] (in /build/pcclassify)
==656== by 0x487CB9D: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==656== by 0x5396B42: start_thread (pthread_create.c:442)
==656== by 0x5427BB3: clone (clone.S:100)
==656== Address 0x15912 is not stack'd, malloc'd or (recently) free'd
==656==
==656==
==656== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==656== Access not within mapped region at address 0x15912
==656== at 0x17F432: rf::classify(PointSet&, liblearning::RandomForest::RandomForest<liblearning::RandomForest::NodeGini<liblearning::RandomForest::AxisAlignedSplitter> >*, std::vector<Feature*, std::allocator<Feature*> > const&, std::vector<Label, std::allocator<Label> > const&, rf::Regularization, bool, bool) [clone ._omp_fn.3] (in /build/pcclassify)
==656== by 0x487CB9D: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==656== by 0x5396B42: start_thread (pthread_create.c:442)
==656== by 0x5427BB3: clone (clone.S:100)
==656== If you believe this happened as a result of a stack
==656== overflow in your program's main thread (unlikely but
==656== possible), you can try to increase the size of the
==656== main thread stack using the --main-stacksize= flag.
==656== The main thread stack size used in this run was 8388608.
==656==
==656== HEAP SUMMARY:
==656== in use at exit: 205,653,345 bytes in 1,223,477 blocks
==656== total heap usage: 16,633,178 allocs, 15,409,701 frees, 1,342,197,731 bytes allocated
==656==
==656== Searching for pointers to 1,223,477 not-freed blocks
==656== Checked 399,635,328 bytes
==656==
==656== LEAK SUMMARY:
==656== definitely lost: 415 bytes in 3 blocks
==656== indirectly lost: 0 bytes in 0 blocks
==656== possibly lost: 11,408 bytes in 23 blocks
==656== still reachable: 205,641,522 bytes in 1,223,451 blocks
==656== suppressed: 0 bytes in 0 blocks
==656== Rerun with --leak-check=full to see details of leaked memory
==656==
==656== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
==656==
==656== 1 errors in context 1 of 1:
==656== Invalid write of size 1
==656== at 0x17F432: rf::classify(PointSet&, liblearning::RandomForest::RandomForest<liblearning::RandomForest::NodeGini<liblearning::RandomForest::AxisAlignedSplitter> >*, std::vector<Feature*, std::allocator<Feature*> > const&, std::vector<Label, std::allocator<Label> > const&, rf::Regularization, bool, bool) [clone ._omp_fn.3] (in /build/pcclassify)
==656== by 0x487CB9D: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==656== by 0x5396B42: start_thread (pthread_create.c:442)
==656== by 0x5427BB3: clone (clone.S:100)
==656== Address 0x15912 is not stack'd, malloc'd or (recently) free'd
==656==
==656== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault
It crashes both on my linux system (kde neon 5.27) and on linux docker image (ubuntu 22.04)
pctrain
by adding a --extract-features <path>.opcfeat.bin
parameter. When set, execution should stop at https://github.com/uav4geo/OpenPointClass/blob/main/randomforest.cpp#L30 and https://github.com/uav4geo/OpenPointClass/blob/main/gbm.cpp#L45gt
, ft
, GBT populates the structures similarly although not identically). It might also be possible to serialize in a single format regardless of RF or GBT if one creates a new function that simply does the serialization (like train
, but stops after creating the features). One might want to encode the various scale
, radius
, treeDepth
, etc. parameters into the serialized output to avoid repeating them and validating other serialized outputs. All serialized output's parameters from different processes need to match.pctrain
by checking for .opcfeat.bin
file input extensions; if all files passed as input are .opcfeat.bin
, then read features directly instead of computing them by adapting the rf::train
and gbt::train
functions. If you have serialized the scale
, radius
, etc. parameters one can read them from the serialized files instead of passing them manually.Hi,
I get a segfault for a .laz file that I am trying to classify using docker. It looks like it fails during the creation of the second pass.
./pcclassify /data/21A027_628_58_0000.laz ./classified.ply /data/model.bin
I am uncertain if it it is a bug, but rather an older LAS specification version or similar that is causing the issue. You have specified that version 1.4 is required for training - is it the same for pcclassify? If this is the case, do you have any recommendation for conversion. I did try to convert it without any success. The location of the laser data is in the southern part of Sweden.
Please let me know if I need to provide you with any additional information.
Output of lasinfo:
Heading '21A027_628_58_0000.laz' with 11317852 points
lasinfo (240220) report for '21A027_628_58_0000.laz'
reporting all LAS header entries:
file signature: 'LASF'
file source ID: 0
global_encoding: 1
project ID GUID data 1-4: 00000000-0000-0000-0000-000000000000
version major.minor: 1.2
system identifier: ''
generating software: 'LASzip DLL 2.4 r0 (150731)'
file creation day/year: 336/2021
header size: 227
offset to point data: 227
number var. length records: 0
point data format: 1
point data record length: 28
number of point records: 11317852
number of points by return: 7718162 1749178 983754 571157 295601
scale factor x y z: 0.01 0.01 0.01
offset x y z: 535000 6715000 0
min x y z: 580000.00 6280000.00 -390.66
max x y z: 582499.99 6282499.99 341.04
LASzip compression (version 2.4r0 c2 50000): POINT10 2 GPSTIME11 2
reporting minimum and maximum for all LAS point record entries ...
X 4500000 4749999
Y -43500000 -43250001
Z -39066 34104
intensity 382 65535
return_number 1 5
number_of_returns 1 5
edge_of_flight_line 0 0
scan_direction_flag 0 1
classification 1 18
scan_angle_rank -18 18
user_data 0 0
point_source_ID 13603 13605
gps_time 300525420.164837 300527088.064599
number of first returns: 7718162
number of intermediate returns: 1849949
number of last returns: 7719279
number of single returns: 5969538
overview over number of returns of given pulse: 5969538 1531136 1237623 1102336 1477219 0 0
histogram of classification of points:
4987784 unclassified (1)
6310052 ground (2)
8978 noise (7)
2528 water (9)
8420 bridge deck (17)
90 Reserved for ASPRS Definition (18)
Output is:
map.ply ./map.ply ./model.bin
Model: Random Forest
Loading ./model.bin
Reading 1496024 points
Starting resolution: 0.01
Init scale 0 at 0.01 ...
Init scale 1 at 0.01 ...
Init scale 2 at 0.02 ...
Init scale 3 at 0.04 ...
Init scale 4 at 0.08 ...
Init scale 5 at 0.16 ...
Building scale 1 (3 points) ...
Building scale 2 (3 points) ...
Building scale 3 (3 points) ...
Building scale 4 (3 points) ...
Building scale 5 (3 points) ...
Features: 105
Classifying...
Local smoothing...
Wrote ./map.ply
#9 0.554 [ 5%] Creating directories for 'lightgbm'
#9 0.619 [ 10%] Performing download step (git clone) for 'lightgbm'
#9 0.637 Cloning into 'lightgbm'...
#9 3.799 HEAD is now at e4231205 [python-package] use keyword arguments in predict() calls (#5755)
#9 3.824 Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'external_libs/compute'
#9 3.825 Submodule 'eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'external_libs/eigen'
#9 3.826 Submodule 'external_libs/fast_double_parser' (https://github.com/lemire/fast_double_parser.git) registered for path 'external_libs/fast_double_parser'
#9 3.826 Submodule 'external_libs/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'external_libs/fmt'
#9 3.831 Cloning into '/opc/build/lightgbm-prefix/src/lightgbm/external_libs/compute'...
#9 5.099 Cloning into '/opc/build/lightgbm-prefix/src/lightgbm/external_libs/eigen'...
#9 5.522 error: RPC failed; HTTP 403 curl 22 The requested URL returned error: 403
#9 5.522 fatal: error reading section header 'shallow-info'
#9 5.525 fatal: clone of 'https://gitlab.com/libeigen/eigen.git' into submodule path '/opc/build/lightgbm-prefix/src/lightgbm/external_libs/eigen' failed
#9 5.530 Failed to clone 'external_libs/eigen'. Retry scheduled
#9 5.534 Cloning into '/opc/build/lightgbm-prefix/src/lightgbm/external_libs/fast_double_parser'...
#9 6.091 Cloning into '/opc/build/lightgbm-prefix/src/lightgbm/external_libs/fmt'...
#9 8.341 Cloning into '/opc/build/lightgbm-prefix/src/lightgbm/external_libs/eigen'...
#9 8.840 error: RPC failed; HTTP 403 curl 22 The requested URL re
Using the Dockerfile, the sha pointed at is no good anymore.
hi dev team,
do you have publication of classification model, can you give me link?
thanks,
i am following the instructions to install. when i use the command
"cmake [-DWITH_GBT=ON] -DCMAKE_TOOLCHAIN_FILE=<C:\Researchteam\vcpkg\vcpkg>/scripts/buildsystems/vcpkg.cmake .."
it show "Access is denied." I already using administrator to open anaconda prompt, please someone help thank you so much.
Hi, when using pctrain with 1.65GB .laz file as an input for training, it it is Starting resolution, jumps to Init scale ....and after short time it outputs Killed. (see attached screen dump). Are there any limitations on file size? Any paramaters we we can adjust to use large sized input files for pctrain?
We are running it as docker image which is deployed on VM servere with Red Hat Enterprise Linux 8.9 (Ootpa) operating system, 64GB RAM, Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (16 cores)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.