SEGV in regression test: Too many open files about vexriscv

[Dolu1990]

Hi,

Hmm, in main.cpp i'm never closing files used in read modes, maybe that's it ?

• https://github.com/SpinalHDL/VexRiscv/blob/master/src/test/cpp/regression/main.cpp#L95
• https://github.com/SpinalHDL/VexRiscv/blob/master/src/test/cpp/regression/main.cpp#L2154

Else, i know it normaly not interfer, but i never tested it with verilator 4.0, always 3.9.

I will submit a fix for the file close.

Regards
Charles

[darkstar007]

OK, let me know if you'd like me to test.

[Dolu1990]

I commited some fix in the dev branch. Can you give a try ?

Thanks :)

[darkstar007]

Just tried it. The good news is that it gets further - the bad news is that it dies with a SEGV at the end of the program. The extra bad news is that turning on debug in the makefile and recompiling makes the SEGV go away! (I always love it when that happens....). I just run the exe through valgrind and will have a look at what its says/means - but there are certainly a number of complaints about " Mismatched free() / delete / delete []" which can cause problems and the final SEGV is caused by/in Workspace::CpuRef::dRead(int, int, unsigned int*).

I shall have a look today and get back to you.

[darkstar007]

OK, some progress. Here is the patch to clean up the warnings from valgrind about the mismatched free/del/del[]:

diff --git a/src/test/cpp/regression/main.cpp b/src/test/cpp/regression/main.cpp
index 016555d..c58ce53 100644
--- a/src/test/cpp/regression/main.cpp
+++ b/src/test/cpp/regression/main.cpp
@@ -35,7 +35,7 @@ public:
                for(uint32_t i = 0;i < (1 << 12);i++) mem[i] = NULL;
        }
        ~Memory(){
-               for(uint32_t i = 0;i < (1 << 12);i++) if(mem[i]) delete mem[i];
+               for(uint32_t i = 0;i < (1 << 12);i++) if(mem[i]) delete [] mem[i];
        }
 
        uint8_t* get(uint32_t address){
@@ -147,7 +147,7 @@ void loadHexImpl(string path,Memory* mem) {
                size--;
        }
 
-       delete content;
+       delete [] content;
 }

The SEGV at the end still occurred after making the above fixes. It turns out that it only happens when the files are compiled with -O3 - if -O0 is used instead then no SEGV is emitted. I'll see if I can compile with gcc-7 instead.

[darkstar007]

OK, so it works fine now with gcc-7 (g++ (Debian 7.3.0-30) 7.3.0) with -O3 turned on. So its a compiler (optimiser) issue?.........

[Dolu1990]

hmm or using another compiler just swap around things in a way the bug isn't showing it's noise but is still there ?
One source of weird behavior could be the tcp serveur which emulate jtag access ?
https://github.com/SpinalHDL/VexRiscv/blob/master/src/test/cpp/regression/main.cpp#L1689

[darkstar007]

Dunno - this is what valgrind says about the SEGV:

CONNECTED
SUCCESS DebugPluginTest
==14645== Invalid read of size 1
==14645==    at 0x114176: Workspace::CpuRef::dRead(int, int, unsigned int*) (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==14645==    by 0x7FFFFFFFFFFFFFFF: ???
==14645==    by 0xFF0000000011CCA7: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==  Address 0x7004333 is 92,339 bytes inside a block of size 215,419 free'd
==14645==    at 0x483758B: operator delete[](void*) (vg_replace_malloc.c:641)
==14645==    by 0x11B2AB: TestX28::TestX28(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned int*, unsigned int) (in /home/matt/github_
cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==14645==    by 0x10E619: main (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==14645==  Block was alloc'd at
==14645==    at 0x483650F: operator new[](unsigned long) (vg_replace_malloc.c:423)
==14645==    by 0x1105B1: loadHexImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Memory*) (in /home/matt/github_cloned/VexRiscv/src/test/
cpp/regression/obj_dir/VVexRiscv)
==14645==    by 0x11B2AB: TestX28::TestX28(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned int*, unsigned int) (in /home/matt/github_
cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==14645==    by 0x10E619: main (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==14645== 
==14645== 
==14645== Process terminating with default action of signal 11 (SIGSEGV)
==14645==  Access not within mapped region at address 0xE004334
==14645==    at 0x114176: Workspace::CpuRef::dRead(int, int, unsigned int*) (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==14645==    by 0x7FFFFFFFFFFFFFFF: ???
==14645==    by 0xFF0000000011CCA7: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==    by 0xFFFFFFFFFFFFFFFE: ???
==14645==  If you believe this happened as a result of a stack
==14645==  overflow in your program's main thread (unlikely but
==14645==  possible), you can try to increase the size of the
==14645==  main thread stack using the --main-stacksize= flag.
==14645==  The main thread stack size used in this run was 8388608.
==14645== 
==14645== HEAP SUMMARY:
==14645==     in use at exit: 5,605,066 bytes in 1,242 blocks
==14645==   total heap usage: 64,520 allocs, 63,278 frees, 5,196,144,996 bytes allocated
==14645== 
==14645== LEAK SUMMARY:
==14645==    definitely lost: 3,478,418 bytes in 1,185 blocks
==14645==    indirectly lost: 1,312 bytes in 3 blocks
==14645==      possibly lost: 2,880 bytes in 10 blocks
==14645==    still reachable: 2,122,456 bytes in 44 blocks
==14645==         suppressed: 0 bytes in 0 blocks
==14645== Rerun with --leak-check=full to see details of leaked memory
==14645== 
==14645== For counts of detected and suppressed errors, rerun with: -v
==14645== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 0 from 0)

Which seems to suggest that its access to RAM which has previously been free'ed. I'm prolly not going to have a chance to chase this until tomorrow now.

[Dolu1990]

Thanks :)
How did you got this access checking messages ? (option) ?

I'm not realy used to it, but if i read correctly, in the function
https://github.com/SpinalHDL/VexRiscv/blob/master/src/test/cpp/regression/main.cpp#L687
There was an byte access done at 0x7004333 which is a memory area which was deleted previously ?
And the another error in the same function with address 0xE004334 which was never allocated ?

[Dolu1990]

I submited a sanity assert about the size of the access, let's me know if it trigger :)

[darkstar007]

I get this by just running 'valgrind ./obj_dir/VVexRiscv', after doing a 'make clean run'

Yep, an illegal access - which is what caused the SEGV (the actual error, rather than the 1st one which is caught by valgrind)

[darkstar007]

Using gcc-8 again I get this:

CONNECTED
SUCCESS DebugPluginTest
dRead size=758f54e8
Segmentation fault

The actual number that prints on the dRead line changes each time I run the program.

If I run it with valgrind, then i get the free/del/del[] warnings which my patch above fixed:

==5060== Mismatched free() / delete / delete []
==5060==    at 0x4836EAB: operator delete(void*) (vg_replace_malloc.c:576)
==5060==    by 0x11AB88: loadHex (main.cpp:778)
==5060==    by 0x11AB88: Compliance::Compliance(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (main.cpp:2140)
==5060==    by 0x10DC68: main (main.cpp:2661)
==5060==  Address 0x4fd4730 is 0 bytes inside a block of size 2,817 alloc'd
==5060==    at 0x483650F: operator new[](unsigned long) (vg_replace_malloc.c:423)
==5060==    by 0x1105B1: loadHexImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Memory*) (main.cpp:107)
==5060==    by 0x11AB88: loadHex (main.cpp:778)
==5060==    by 0x11AB88: Compliance::Compliance(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (main.cpp:2140)
==5060==    by 0x10DC68: main (main.cpp:2661)
==5060== 
==5060== Mismatched free() / delete / delete []
==5060==    at 0x4836EAB: operator delete(void*) (vg_replace_malloc.c:576)
==5060==    by 0x11ABCE: loadHex (main.cpp:779)
==5060==    by 0x11ABCE: Compliance::Compliance(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (main.cpp:2140)
==5060==    by 0x10DC68: main (main.cpp:2661)
==5060==  Address 0x52b9080 is 0 bytes inside a block of size 2,817 alloc'd
==5060==    at 0x483650F: operator new[](unsigned long) (vg_replace_malloc.c:423)
==5060==    by 0x1105B1: loadHexImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Memory*) (main.cpp:107)
==5060==    by 0x11ABCE: loadHex (main.cpp:779)
==5060==    by 0x11ABCE: Compliance::Compliance(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (main.cpp:2140)
==5060==    by 0x10DC68: main (main.cpp:2661)

at the end of the valgrind output it says:

dRead size=fefef078
==5060== Invalid read of size 1
==5060==    at 0x11418A: read (main.cpp:57)
==5060==    by 0x11418A: Workspace::CpuRef::dRead(int, int, unsigned int*) (main.cpp:702)
==5060==    by 0x7FFFFFFFFFFFFFFF: ???
==5060==    by 0xFF0000000011CD07: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==  Address 0x7004333 is 92,339 bytes inside a block of size 215,419 free'd
==5060==    at 0x4836EAB: operator delete(void*) (vg_replace_malloc.c:576)
==5060==    by 0x11B30B: loadHex (main.cpp:779)
==5060==    by 0x11B30B: TestX28::TestX28(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned int*, unsigned int) (main.cpp:2030)
==5060==    by 0x10E619: main (main.cpp:2734)
==5060==  Block was alloc'd at
==5060==    at 0x483650F: operator new[](unsigned long) (vg_replace_malloc.c:423)
==5060==    by 0x1105B1: loadHexImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Memory*) (main.cpp:107)
==5060==    by 0x11B30B: loadHex (main.cpp:779)
==5060==    by 0x11B30B: TestX28::TestX28(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned int*, unsigned int) (main.cpp:2030)
==5060==    by 0x10E619: main (main.cpp:2734)
==5060== 
==5060== 
==5060== Process terminating with default action of signal 11 (SIGSEGV)
==5060==  Access not within mapped region at address 0xE004334
==5060==    at 0x11418A: read (main.cpp:57)
==5060==    by 0x11418A: Workspace::CpuRef::dRead(int, int, unsigned int*) (main.cpp:702)
==5060==    by 0x7FFFFFFFFFFFFFFF: ???
==5060==    by 0xFF0000000011CD07: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==    by 0xFFFFFFFFFFFFFFFE: ???
==5060==  If you believe this happened as a result of a stack
==5060==  overflow in your program's main thread (unlikely but
==5060==  possible), you can try to increase the size of the
==5060==  main thread stack using the --main-stacksize= flag.
==5060==  The main thread stack size used in this run was 8388608.
==5060== 
==5060== HEAP SUMMARY:
==5060==     in use at exit: 5,605,066 bytes in 1,242 blocks
==5060==   total heap usage: 64,520 allocs, 63,278 frees, 5,196,144,996 bytes allocated
==5060== 
==5060== LEAK SUMMARY:
==5060==    definitely lost: 3,478,418 bytes in 1,185 blocks
==5060==    indirectly lost: 1,312 bytes in 3 blocks
==5060==      possibly lost: 2,880 bytes in 10 blocks
==5060==    still reachable: 2,122,456 bytes in 44 blocks
==5060==         suppressed: 0 bytes in 0 blocks
==5060== Rerun with --leak-check=full to see details of leaked memory
==5060== 
==5060== For counts of detected and suppressed errors, rerun with: -v
==5060== ERROR SUMMARY: 7034 errors from 51 contexts (suppressed: 0 from 0)

[Dolu1990]

Ok, thanks !
I will give a try with valgrind :)

[Dolu1990]

(on my config + travis regretion, everything is going well, but i remember already having similar issues a long time ago)

[Dolu1990]

Fixed the deletes, but normaly it will not change the crash.

[Dolu1990]

The things which is realy weird is how dRead size=758f54e8 is possible ?
If we look the dRead functions call, it normaly can not be outside 1 to 8

[darkstar007]

I put many 'cout << LINE << end;' to see how the program flowed. This changed the crash. For example I put in a line like the above just before any calls to dRead, and none of them were printed out - so there is no explicit call into the dRead function, its just falling into that bit of memory?.....

Its all very strange. With the current set of cout's I can see that when doing dhrystoneO3_Stall :

403:   	iRead(pc, &i);  

(pc = 0x80000000) then

683:          mem.read(address, 4, (uint8_t*)data);

now *data = 0xb00006f

897:    virtual void iBusAccessPatch(uint32_t addr, uint32_t *data, bool *error){}

Now it prints 'decoding error':

decoding error
==25044== Invalid read of size 8
==25044==    at 0x113CCE: RiscvGolden::decodingError() (main.cpp:314)
==25044==    by 0x8000000000000000: ???
==25044==    by 0x4DAF75F: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==25044==    by 0x4DAB29F: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==25044==    by 0x1FFEFF795F: ???
==25044==    by 0x14F4BF: ??? (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==25044==    by 0x14F4BF: ??? (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==25044==    by 0x7FFFFFFFFFFFFFFF: ???
==25044==    by 0x1FFEFEF0CE: ???
==25044==    by 0x2DC6BF: ???
==25044==    by 0x114BD0: RiscvGolden::step() (main.cpp:410)
==25044==    by 0x116B03: Workspace::CpuRef::step() (main.cpp:739)
==25044==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==25044== 
==25044== 
==25044== Process terminating with default action of signal 11 (SIGSEGV)
==25044==  Access not within mapped region at address 0x0
==25044==    at 0x113CCE: RiscvGolden::decodingError() (main.cpp:314)
==25044==    by 0x8000000000000000: ???
==25044==    by 0x4DAF75F: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==25044==    by 0x4DAB29F: ??? (in /lib/x86_64-linux-gnu/libc-2.27.so)
==25044==    by 0x1FFEFF795F: ???
==25044==    by 0x14F4BF: ??? (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==25044==    by 0x14F4BF: ??? (in /home/matt/github_cloned/VexRiscv/src/test/cpp/regression/obj_dir/VVexRiscv)
==25044==    by 0x7FFFFFFFFFFFFFFF: ???
==25044==    by 0x1FFEFEF0CE: ???
==25044==    by 0x2DC6BF: ???
==25044==    by 0x114BD0: RiscvGolden::step() (main.cpp:410)
==25044==    by 0x116B03: Workspace::CpuRef::step() (main.cpp:739)

So in my code around line 403 it looks like:

		    std::cout << __LINE__ << "  " << hex << pc << dec << std::endl;
			iRead(pc, &i);
		std::cout << __LINE__ << std::endl;

And the second cout is never printed! Stack corruption?

Oh, and by the way on line 1116 you have:

cout << "FAIL " <<  name << " at PC=" << hex << setw(8) << top->VexRiscv->writeBack_PC << endl;

this doesn't set the stream back into 'dec' mode which made all following cout's for my line numbers print in hex which caused some confusion!!!

[Dolu1990]

Hmm, maybe in can try to reproduce the bug by using a similar system configuration than your one ?

What is or OS / gcc version (where the bug happen the more often) ?

Also, how much stack is allocated per thread on your machine ?
(ulimit -s)

Sorry for that mess XD
I sibmited a fix for the hex dec stuff.

[Dolu1990]

Hooo on the top "This is with Verilator 4.006 2018-10-27 and gcc version 8.2.0"
I will try with the exact same setting

[Dolu1990]

Ok, updated only GCC to 8.1 and got the same issue :D

[Dolu1990]

The dhrystone benchs specificaly make the thing crashing
Commenting out the golden model avoid the crash (//withRiscvRef(); in the Dhrystone constructor)

[Dolu1990]

Ok, i finaly got it, the issue was that some virtual function with a return type were not returning a value (missing return statement, even if that returned value was never used)

I didn't know it could cause the program to corrupt and crash XD

Thanks for your help :D