ERROR: AddressSanitizer: new-delete-type-mismatch when using AWS but not when using GCP about openradioss

[schrummy14]

Hello,
Thank you for the update. I'll look into the limits and see what I need to do on that end.

With the latest update (2024-03-01) both of the examples are now working.

[schrummy14]

Running the Bumper-Beam example rad file ends in a seg-fault.

 .. RETURNS TO DOMAIN DECOMPOSITION FOR OPTIMIZATION
 .. DOMAIN DECOMPOSITION
 .. ELEMENT GROUPS                                                                       
 .. INTERFACES                                                                           
 .. INTERFACE BUFFER INITIALIZATION                                                      

WARNING ID :    343
** WARNING: INITIAL PENETRATIONS IN INTERFACE
 .. RIGID WALLS                                                                          
 .. RIGID BODIES                                                                         
 .. ELEMENT BUFFER INITIALIZATION                                                        
 .. SECTIONS
Segmentation fault

[MarianBulla]

Hello Matt,
thank you for your interest on OpenRadioss.
Yes, we are able to reproduce this issue and we are working on it, now. Good catch ! Thank you very much for reporting it.
Very sorry about that.

Anyway, there is a workaround:

• switch the sanitizer OFF: "export ASAN_OPTIONS=new_delete_type_mismatch=0:detect_leaks=0:halt_on_error=1"
• Compile the executables again with the option "-release"

Hope it helps you ?

Thanks and best regards,
Marian

[schrummy14]

Hello,
Thank you for the information, but it is throwing a seg fault error.

Steps to build the starter:

export PATH=/opt/JARVICE/bin:/opt/JARVICE/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/opt/JARVICE/lib:/opt/JARVICE/openmpi/lib
export OPENRADIOSS_PATH=/opt/OpenRadioss
export RAD_CFG_PATH=$OPENRADIOSS_PATH/hm_cfg_files
export RAD_H3D_PATH=$OPENRADIOSS_PATH/extlib/h3d/lib/linux64
export OMP_STACKSIZE=1024m
export LD_LIBRARY_PATH=$OPENRADIOSS_PATH/extlib/hm_reader/linux64/:$LD_LIBRARY_PATH
export ASAN_OPTIONS="new_delete_type_mismatch=0:detect_leaks=0:halt_on_error=1"
cd /opt/OpenRadioss/starter/
/opt/OpenRadioss/starter/build_script.sh -arch=linux64_gf -release -prec=dp -nt=96
cd /data/openradioss/Bumper_Beam/
/opt/OpenRadioss/exec/starter_linux64_gf -i Bumper_Beam_AP_meshed_0000.rad -nt 96 -np 96

Output:

************************************************************************
**                                                                    **
**                                                                    **
**                        OpenRadioss Starter                         **
**                                                                    **
**            Non-linear Finite Element Analysis Software             **
**                                                                    **
**                                                                    **
**                                                                    **
**                    Linux 64 bits, GNU compiler                     **
**                                                                    **
**                                                                    **
**                                                                    **
**                                                                    **
************************************************************************
** OpenRadioss Software                                               **
** COPYRIGHT (C) 1986-2024 Altair Engineering, Inc.                   **
** Licensed under GNU Affero General Public License.                  **
** See License file.                                                  **
************************************************************************
 

 .. UNITS SYSTEM                                                                         
 .. CONTROL VARIABLES                                                                    
 .. STARTER RUNNING ON   96 THREADS

WARNING ID :   1032
** WARNING STACKSIZE LIMIT IS LOWER THAN 512M

 .. FUNCTIONS & TABLES
 .. MATERIALS                                                                            
 .. NODES                                                                                
 .. PROPERTIES                                                                           
 .. 3D SHELL ELEMENTS                                                                    
 .. 3D SPRING ELEMENTS                                                                   
 .. 3D TRIANGULAR SHELL ELEMENTS                                                         
 .. SUBSETS
 .. ELEMENT GROUPS
 .. PART GROUPS
 .. SURFACES 
 .. NODE GROUP
 .. BOUNDARY CONDITIONS                                                                  
 .. INITIAL VELOCITIES                                                                   
 .. ACCELEROMETERS
 .. DOMAIN DECOMPOSITION
 .. ELEMENT GROUPS                                                                       
 .. INTERFACES                                                                           
 .. INTERFACE BUFFER INITIALIZATION                                                      
 .. RIGID WALLS                                                                          
 .. RIGID BODIES                                                                         
 .. RETURNS TO DOMAIN DECOMPOSITION FOR OPTIMIZATION
 .. DOMAIN DECOMPOSITION
 .. ELEMENT GROUPS                                                                       
 .. INTERFACES                                                                           
 .. INTERFACE BUFFER INITIALIZATION                                                      

WARNING ID :    343
** WARNING: INITIAL PENETRATIONS IN INTERFACE
 .. RIGID WALLS                                                                          
 .. RIGID BODIES                                                                         
 .. ELEMENT BUFFER INITIALIZATION                                                        
 .. SECTIONS
Segmentation fault (core dumped)

I am not sure if this is a direct result of Amazons kernel or not as both the GCP and AWS use 5.10.186, but Amazon does add to it.
AWS Machine (AMD EPYC 7R13 Processor):

uname -a
Linux jarvice-job-104519-zh749 5.10.186-179.751.amzn2.x86_64 #1 SMP Tue Aug 1 20:51:38 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Local Machine (AMD Ryzen 9 5950X 16-Core Processor):

uname -a
Linux fedora 6.6.13-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Jan 20 18:03:28 UTC 2024 x86_64 GNU/Linux

GCP machine (Intel(R) Xeon(R) CPU @ 3.10GHz, CPU family: 6, Model: 85):

uanme -a
Linux jarvice-job-104525-lx6xn 5.10.186+ #1 SMP Sat Sep 23 15:15:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[elequiniou]

Hi, I am not sure the error why you got such error, but I noticed you request 96 threads. This is not optimal and you don't need it. I would advise to set -nt 1 to be sure there is no issue with OpenMP stack size (keep -np 96 to be able to run over 96 MPI the Engine executable). Please, let us know if it solves?
FYI, the -nt for starter only affects the improvement of domain splitting at the end of the starter, I would advise to keep it to 4 or 8 threads (this will help with the writing of restart files by allowing parallel writing under OpenMP). Thanks.

[schrummy14]

Hello,
Yes, I have tried with -nt 1 -np 1 but it also yields a seg fault error.

[OlivierWienholtz]

Hello,

Recently we have changed the build defaults of Radioss. Now the debault executable under Linux contains the address Sanitizer.
We see that you are facing 2 issues : one with the "tensile.k" model and "Bumper_Beam_AP_meshed"

The first one, Tensile.k" is related to the address Sanitizer change.
This one should go through with the set of flags Marian gave you :
export ASAN_OPTIONS=new_delete_type_mismatch=0:detect_leaks=0:halt_on_error=1

I tried the tensile Dyna model from our OpenRadioss examples : (zug_test3_RS).
I reproduced the issue you had when ASAN_OPTIONS is not set.
It is running through when ASAN Option is set.

You second issue with Example model Bumper_Beam_AP_meshed_0000.rad
I did not reproduce your issue for now. I tried several issues, including an AMD machine, & several domain decomposition flags.

Best Regards,
Olivier

[OlivierWienholtz]

Is the model you have taken, the one from OpenRadioss confluence ?
Have yo made any change in ?

Thank You

[schrummy14]

Tensile Test: https://openradioss.atlassian.net/wiki/spaces/OPENRADIOSS/pages/21364914/Tensile+Test+Model+in+LS-DYNA+format

Bumper-Beam https://openradioss.atlassian.net/wiki/spaces/OPENRADIOSS/pages/11075585/Bumper+Beam

No changes to either. Using the env variable: ASAN_OPTIONS=new_delete_type_mismatch=0:detect_leaks=0:halt_on_error=1
along with -release during building, both tests yield seg fault errors.

The seg fault happens on both the tensile and the bumper_beam input scripts if I use the pre-packed version of OpenRadioss downloaded from the release page (just tested with the latest version https://github.com/OpenRadioss/OpenRadioss/releases/tag/latest-20240129).

[OlivierWienholtz]

Hi Matt,

I used same models.
What compiler have you used to build OpenRadioss ?
Can you download an OpenRadioss release : https://github.com/OpenRadioss/OpenRadioss/releases/tag/latest-20240129
& check if this one works ?

We are using Devtoolset-11 (gcc/gfortran 11.2.1) to build OpenRadioss.

Thank You !
Best Regards,
Olivier

[schrummy14]

Hello,
Yes, I did try the latest pre-compiled version. This also caused the seg fault errors.
gcc/gfortran versions are 11.4.1 from Rocky Linux 9

gcc --version
gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

gfortran --version
GNU Fortran (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

OS-Info

cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

[laurent-altr]

Hello @schrummy14 ,
I have not been able to reproduce the error that occurred during the processing of sections. Could you please provide us with the results of the command 'ulimit -a'?

[schrummy14]

Hello,
Please find the results below for local, GCP, and AWS
local:

AMD Ryzen 9 5950X 16-Core Processor
Linux fedora 6.6.14-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jan 26 20:12:16 UTC 2024 x86_64 GNU/Linux
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 127645
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 127645
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

GCP:

  Model name:            Intel(R) Xeon(R) CPU @ 3.10GHz
    CPU family:          6
    Model:               85
Linux jarvice-job-107131-sbcwr 5.10.186+ #1 SMP Sat Sep 23 15:15:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 967210
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

AWS:

AMD EPYC 7R13 Processor
Linux jarvice-job-107129-lg89m 5.10.186-179.751.amzn2.x86_64 #1 SMP Tue Aug 1 20:51:38 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 30446
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 10240
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

[OlivierWienholtz]

Hello Matt

Just some informations :

• The build defaults are now back to optimized. (eg -release).
• Some fixes have been done with the Starter & dynae decks when running with address sanitizer.
• The limits looks ok. The models are quite small 8MB of stacksize should be sufficient on your bare metal AMD. However, trying to set it to unlimited could be good as bigger models could have issues...

I am wondering if you made some progress on your issues or if you are still blocked.

Thank You
Olivier W.

[schrummy14]

Hello,
Using the latest release and the build command:

export ASAN_OPTIONS="new_delete_type_mismatch=0:detect_leaks=0:halt_on_error=1"
EXTRA_ARGS="-mtune=generic -march=x86-64-v3"
NUM_THREADS=16
./build_script.sh -arch=linux64_gf -nt="${NUM_THREADS}" -prec=dp -debug=0 -release -addflag="${EXTRA_ARGS}"

Yeilds the same seg fault.

Unfortunately, I have not had time look further into this. Would you be able to provide a general area that I should search into? There is currently no blocker.

[OlivierWienholtz]

Hello,

I found out that Stack usage for Starter is about 11MB which is little higher than your Bare Metal EPYC...
You may have hardcoded Stack limits.

You can set in :
/etc/security/limits.conf

• soft memlock unlimited
• hard memlock unlimited
• soft stack unlimited
• hard stack unlimited
• soft nofile 16384
• hard nofile 16384

and
ulimit -s unlimited in your .bashrc...

That should solve.

In the mean time, I found some excessive Stack usage in one option.
I'll fix this in the next days.

[OlivierWienholtz]

Hello
with 1f16ff0
we have reduced the Stacksize usage.
The Bumper Beam should run to completion now.

Let me know if this works for you.
Regards,
Olivier W.

[OlivierWienholtz]

Great to hear that.
I'll close the issue.
Do not hesitate to reach us out again if you have issues again.