Global task_schedule_observer leads to error on WASM and static lib x86 about onetbb

[WardBrian]

#1346 leads to a bunch of extra compiler warnings about aligned-new, but it resolves the issue with em++ and for g++ + static libs

[WardBrian]

Compiling with optimizations -O3 yields an even more cryptic error when run by node:

RuntimeError: null function or function signature mismatch
    at wasm://wasm/000993ba:wasm-function[115]:0x948c
    at wasm://wasm/000993ba:wasm-function[50]:0x49ac
    at wasm://wasm/000993ba:wasm-function[32]:0x36d7
    at wasm://wasm/000993ba:wasm-function[95]:0x7e00
    at callRuntimeCallbacks (/home/brian/Dev/cpp/tinystan/test.js:1:7797)
    at initRuntime (/home/brian/Dev/cpp/tinystan/test.js:1:3877)
    at doRun (/home/brian/Dev/cpp/tinystan/test.js:1:15378)
    at run (/home/brian/Dev/cpp/tinystan/test.js:1:15657)
    at runCaller (/home/brian/Dev/cpp/tinystan/test.js:1:14784)
    at removeRunDependency (/home/brian/Dev/cpp/tinystan/test.js:1:4829)

Node.js v19.9.0

[SoilRos]

Something is very fishy here. The static variable controls seems to have a segfault. I did not understand why but I suspect is about the initialization order.

oneTBB/src/tbb/global_control.cpp

Line 146 in 224eca3

static control_storage *controls[] = {&allowed_parallelism_ctl, &stack_size_ctl, &terminate_on_exception_ctl, &lifetime_ctl};

When this is fixed (e.g. storing its contents on the heap), the next bug is even more cryptic to me: The stack base and the thread stack size fail in the assertion below when invoking initializing a thread in init_external_thread():

oneTBB/src/tbb/scheduler_common.h

Line 570 in 224eca3

__TBB_ASSERT(base > stack_size / 2, "Stack anchor calculation overflow");

This is far from my understanding. But I suspect that this comes from the hardcoded values for the stack size:

oneTBB/src/tbb/misc.h

Line 61 in 224eca3

const std::size_t ThreadStackSize = (sizeof(uintptr_t) <= 4 ? 2 : 4 )*MByte;

and maybe a wrong assumption on the trick to get the stack base address:

oneTBB/src/tbb/governor.cpp

Line 118 in 224eca3

There is no portable way to get stack base address in Posix, however the modern

This happens also when the observer is disabled. Meaning that the segfault reported here may be actually hiding a bug on the thread initialization...

[pavelkumbrasev]

I believe @SoilRos is right here and there is problem with order of global objects initialization i.e., both objects control_storage and global observer are global.
We had similar problem in past that were triggered with allocator initialization but since we fixed there were no reports.
I'm not sure if it's WASM specific (I could not reproduce it with gcc on Ubuntu).
@WardBrian Could you work-around this issue by calling observe in the start of the main?

[pavelkumbrasev]

@SoilRos could you please describe what example are you running to get new error?

[WardBrian]

Hi @pavelkumbrasev -

The example I provided is simplified from one in the Stan Math library. It would be non-trivial to move this to main, if only because there are multiple users of the library who would all need to do such a move. If there is no other option for wasm I could likely do some #ifdef trickery.

[SoilRos]

@pavelkumbrasev The docker file in https://github.com/josephholten/em-multi/tree/tbb-1314 (branch tbb-1341) reproduces the fix (using #1346) and the new error stated above on. It basically runs a scalar product on a TBB parallel_for and adds an observer to the scheduler. When built in release mode, the assert is not checked and seems to run. However, the observers are called after the parallel loop. I tried give the thread initialization more time, but it did not work. It may be related to the thread initialization problems I stated above.

TBB thread index: 0
C++ thread: 130372
filling random vectors... 67108864
calculating sequential scalarproduct...
using thread: 130372
seq scalarprod: 16775473.896405
seq time: 48ms
calculating tbb_threads scalarproduct...
using thread: 130372
tbb_threads scalarprod: 32788.941543
tbb_threads time: 57ms
TBB thread index: 1
C++ thread: 1073994376
TBB thread index: 2
C++ thread: 1076093760
TBB thread index: 3
C++ thread: 1078196232

[pavelkumbrasev]

@SoilRos init_external_thread in that simplified scenario will be called only for main thread so it should not affect workers.
It's also possible that thread creation takes more time then parallel region. You can try to set thread barrier + tbb::static_partitioner to make sure that worker threads will eventually arrive into parallel region.

Regarding the main issue. I'm not sure right now if this issue related to TBB or can be reproduced with any other library if order of global objects initialization is involved. If the problem is persistent everywhere then the right thing is report it to em++.

[WardBrian]

@pavelkumbrasev I was able to reproduce with g++ when I built TBB as a static library (which em++ always does)

cmake .. -DTBB_STRICT=OFF -DTBB_DISABLE_HWLOC_AUTOMATIC_SEARCH=ON -DBUILD_SHARED_LIBS=OFF -DTBB_EXAMPLES=OFF -DTBB_TEST=OFF
cmake --build . -j10
g++ -o test test.cpp -I../include -Lgnu_11.4_cxx11_64_relwithdebinfo/ -ltbb

output:

Program received signal SIGSEGV, Segmentation fault.
tbb::detail::r1::global_control_active_value_unsafe (param=param@entry=tbb::detail::d1::global_control::max_allowed_parallelism) at /home/brian/Dev/cpp/oneTBB/src/tbb/global_control.cpp:163
163	    return controls[param]->active_value_unsafe();