Crashes on fishtest about fairy-stockfish

[ianfab]

@ppigazzini Sorry if my comment was misleading, it is expected that the windows workers still crash with the currently running tests since they are using versions before the fix, so only after those tests are finished and new tests with versions after the fix are started, no crashes are expected any more. I will submit new tests with the fixed versions in the evening (CET).

However, now that you mentioned mingw/gcc on windows I realized that this case presumably is not covered yet, since my changes only increase stack size for compilation with MSVC as well as on macOS. Maybe you could try to modify the custom_make.txt of your worker and explicitly pass the stack size, e.g., using -Wl,--stack,8388608 as an argument to g++. I should also cover this scenario in the makefile.

[ppigazzini]

@ianfab

• fairy-stockfish-2e14523.exe (built from mingw_stack_size 2e14523) finished successfully asean matches (tested twice, parameters copied from a test run):

PS D:\__test_worker\mv> .\cutechess-cli.exe -repeat -rounds 100 -tournament gauntlet -resign movecount=8 score=800 -draw movenumber=34 movecount=8 score=20 -concurrency 7 -openings file=asean.epd format=epd order=random plies=16 -variant asean -engine cmd=fairy-stockfish-2e14523.exe option.Hash=4 -engine cmd=fairy-stockfish-2e14523.exe option.Hash=4 -each proto=uci tc=49.85+0.50 option.Threads=1
Indexing opening suite...
Warning: 2 opening repetitions vs 1 games per encounter
Started game 1 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 3 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
(...)
Finished game 99 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): 0-1 {Black wins by adjudication}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 27 - 16 - 57  [0.555] 100
Elo difference: 38.4 +/- 44.8, LOS: 95.3 %, DrawRatio: 57.0 %
Finished match

• fairy-stockfish.exe (built from master 900e95d) disconnect immediately in asean matches (tried 5 times):

PS D:\__test_worker\mv> .\cutechess-cli.exe -repeat -rounds 100 -tournament gauntlet -resign movecount=8 score=800 -draw movenumber=34 movecount=8 score=20 -concurrency 7 -openings file=asean.epd format=epd order=random plies=16 -variant asean -engine cmd=fairy-stockfish.exe option.Hash=4 -engine cmd=fairy-stockfish.exe option.Hash=4 -each proto=uci tc=49.85+0.50 option.Threads=1
Indexing opening suite...
Warning: 2 opening repetitions vs 1 games per encounter
Started game 1 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 2 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 3 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 4 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 5 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 6 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Started game 7 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Finished game 2 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): 1/2-1/2 {Draw by adjudication}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 0 - 1  [0.500] 1
Started game 8 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Finished game 1 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): 1/2-1/2 {Draw by adjudication}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 0 - 2  [0.500] 2
Started game 9 of 100 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT)
Terminating process of engine Fairy-Stockfish 260819 64 POPCNT(4)
Finished game 3 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): 0-1 {White disconnects}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Finished game 8 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): * {No result}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Finished game 5 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): * {No result}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Finished game 4 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): * {No result}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Finished game 9 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): * {No result}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Finished game 6 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): * {No result}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Finished game 7 (Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT): * {No result}
Score of Fairy-Stockfish 260819 64 POPCNT vs Fairy-Stockfish 260819 64 POPCNT: 0 - 1 - 2  [0.333] 3
Elo difference: -120.4 +/- 263.9, LOS: 15.9 %, DrawRatio: 66.7 %
Finished match

• fairy-stockfish-2e14523.exe snippet build log:

make profile-build ARCH=x86-64-modern COMP=mingw -j
(...)
g++ -Wall -Wcast-qual -fno-exceptions -std=c++11 -fprofile-use -fno-peel-loops -fno-tracer -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT   -c -o syzygy/tbprobe.o syzygy/tbprobe.cpp
g++ -o stockfish.exe benchmark.o bitbase.o bitboard.o endgame.o evaluate.o main.o material.o misc.o movegen.o movepick.o pawns.o piece.o position.o psqt.o search.o thread.o timeman.o tt.o uci.o ucioption.o variant.o syzygy/tbprobe.o -lgcov -static -Wl,--stack,8388608
(...)

• fairy-stockfish.exe snippet build log:

make profile-build ARCH=x86-64-modern COMP=mingw -j
(...)
g++ -Wall -Wcast-qual -fno-exceptions -std=c++11 -fprofile-use -fno-peel-loops -fno-tracer -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT   -c -o syzygy/tbprobe.o syzygy/tbprobe.cpp
g++ -o stockfish.exe benchmark.o bitbase.o bitboard.o endgame.o evaluate.o main.o material.o misc.o movegen.o movepick.o pawns.o piece.o position.o psqt.o search.o thread.o timeman.o tt.o uci.o ucioption.o variant.o syzygy/tbprobe.o -lgcov -static
(...)

[ianfab]

After the merge of the fix from official stockfish, I reverted my workaround, since it should not be required any more.

[ianfab]

While searching for a possible cause I fixed a bug in 05d1624, but that should be unrelated to this issue. The issue itself is still difficult to reproduce.

[ianfab]

Testing suggests that these crashes could be stack overflows caused by the increased stack size from the extension of the move list in 9633261, see this failing build in appveyor.

I will push a commit to increase stack size for windows and macOSX. @ppigazzini This should hopefully also fix the crashes on your windows worker.

[ppigazzini]

@ianfab Let me know if you need some tests on Windows.

[ianfab]

@ppigazzini Thanks, since testing with appveyor clearly showed that stack size was the issue, I consider this to be resolved. I will let you know when all tests of the problematic versions have finished on fishtest, since after that windows workers should run stably again. Then it would be nice if you could add back the worker for a while just to confirm that this is the case.

[ppigazzini]

@ianfab the windows worker is collecting some crashes:
http://35.161.250.236:6543/tests/view/5d565f346e23db34f4206d9d
http://35.161.250.236:6543/tests/view/5d546ebf6e23db34f4206d97
http://35.161.250.236:6543/tests/view/5d595a136e23db247f6b64ff

I'm using g++ built by MinGW-W64 project to avoid this msys2 MinGW-W64 problem

g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=C:/msys64/mingw64-810/bin/../libexec/gcc/x86_64-w64-mingw32/8.1.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-8.1.0/configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysroot=/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64 --enable-shared --enable-static --disable-multilib --enable-languages=c,c++,fortran,lto --enable-libstdcxx-time=yes --enable-threads=posix --enable-libgomp --enable-libatomic --enable-lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --enable-version-specific-runtime-libs --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootstrap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 --with-libiconv --with-system-zlib --with-gmp=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-mpfr=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-mpc=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mingw810/prerequisites/x86_64-w64-mingw32-static --with-pkgversion='x86_64-posix-seh-rev0, Built by MinGW-W64 project' --with-bugurl=https://sourceforge.net/projects/mingw-w64 CFLAGS='-O2 -pipe -fno-ident -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2 -pipe -fno-ident -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' CPPFLAGS=' -I/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/include -I/c/mingw810/prerequisites/x86_64-zlib-static/include -I/c/mingw810/prerequisites/x86_64-w64-mingw32-static/include' LDFLAGS='-pipe -fno-ident -L/c/mingw810/x86_64-810-posix-seh-rt_v6-rev0/mingw64/opt/lib -L/c/mingw810/prerequisites/x86_64-zlib-static/lib -L/c/mingw810/prerequisites/x86_64-w64-mingw32-static/lib '
Thread model: posix
gcc version 8.1.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project)

[ianfab]

@ppigazzini How does your make command look like? I noticed that it is actually not easy to inject the additional argument via the make command without overriding other flags.

2e14523 should address the issue. It would be nice if you could run two comparable matches with and without the changed makefile to check whether it fixes the crashes. From the tests on fishtest it seems that the variant does not matter much, but for hoppelpoppel and asean the crashes were most frequent.

[ppigazzini]

@ianfab I'm overwriting the CXXFLAGS:

make profile-build ARCH=x86-64-modern COMP=mingw CXXFLAGS='-Wall -Wcast-qual -fno-exceptions -std=c++11 -fprofile-generate -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -Wl,--stack,8388608' -j

I will test 2e14523 in the evening, some (real) homekeeping to do now :)

[ianfab]

@ppigazzini Thanks, the results sound quite conclusive, so I will merge it.

I just wonder whether there is a better way to maintain the currently required stack size than having it in three different places of the code for macOS, MSVC, and mingw. Maybe making it a default value in the makefile would be an option, but since I anyway do not expect any further change any time soon, it does not really matter much.