pmodels / bolt

Official BOLT Repository

BOLT: OpenMP over Lightweight Threads

BOLT targets a high-performing OpenMP implementation, especially specialized for fine-grain parallelism. BOLT utilizes a lightweight threading model for its underlying threading mechanism. It currently adopts Argobots, a new holistic, low-level threading and tasking runtime, in order to overcome shortcomings of conventional OS-level threads. The current BOLT implementation is based on the OpenMP runtime in LLVM, and thus it can be used with LLVM/Clang, Intel OpenMP compiler, and GCC. More information about BOLT can be found at http://www.bolt-omp.org.

1. Getting Started
2. Testing BOLT
3. BOLT-Specific Environmental Variables
4. Reporting Problems
5. Alternate Build Options

---

1. Getting Started ==================

The following instructions take you through a sequence of steps to get the default configuration of BOLT up and running.

Henceforth, VERSION indicates the version number of the release tarball.

(a) You will need the following prerequisites.

- REQUIRED: This tar file bolt-VERSION.tar.gz

- REQUIRED: C and C++ compilers (gcc and g++ are sufficient)

- REQUIRED: CMake (http://www.cmake.org/download)

- OPTIONAL: Argobots (http://www.argobots.org)
            The BOLT release tarball includes the Argobots source code, and
            thus you can build BOLT together with the built-in Argobots.
            Of course, you can use your own Argobots build instead of the
            accompanied one.  In the latter case, we assume Argobots has
            been installed in /home/USERNAME/argobots-install.

Also, you need to know what shell you are using since different shell has different command syntax. Command "echo $SHELL" prints out the current shell used by your terminal program.

Note: if you obtained BOLT via github, the following commands download the built-in Argobots from the Argobots repository.

git submodule init
git submodule update

(b) Unpack the tar file and create a build directory:

tar xzf bolt-VERSION.tar.gz
mkdir bolt-build
cd bolt-build

If your tar doesn't accept the z option, use

gunzip bolt-VERSION.tar.gz
tar xf bolt-VERSION.tar
mkdir bolt-build
cd bolt-build

(c) Choose an installation directory, say /home/USERNAME/bolt-install, which is assumed to be non-existent or empty.

(d) Configure BOLT specifying the installation directory:

If you want to use the built-in Argobots,

for csh and tcsh:

  cmake ../bolt-VERSION -G "Unix Makefiles" \
      -DCMAKE_INSTALL_PREFIX=/home/USERNAME/bolt-install \
      -DCMAKE_C_COMPILER=<C compiler> \
      -DCMAKE_CXX_COMPILER=<C++ compiler> \
      -DOPENMP_TEST_C_COMPILER=<C compiler for testing> \
      -DOPENMP_TEST_CXX_COMPILER=<C++ compiler for testing> \
      -DCMAKE_BUILD_TYPE=Release \
      -DLIBOMP_USE_ARGOBOTS=on \
      |& tee c.txt

for bash and sh:

  cmake ../bolt-VERSION -G "Unix Makefiles" \
      -DCMAKE_INSTALL_PREFIX=/home/USERNAME/bolt-install \
      -DCMAKE_C_COMPILER=<C compiler> \
      -DCMAKE_CXX_COMPILER=<C++ compiler> \
      -DOPENMP_TEST_C_COMPILER=<C compiler for testing> \
      -DOPENMP_TEST_CXX_COMPILER=<C++ compiler for testing> \
      -DCMAKE_BUILD_TYPE=Release \
      -DLIBOMP_USE_ARGOBOTS=on \
      2>&1 | tee c.txt

If you want to use your own Argobots build,

for csh and tcsh:

  cmake ../bolt-VERSION -G "Unix Makefiles" \
      -DCMAKE_INSTALL_PREFIX=/home/USERNAME/bolt-install \
      -DCMAKE_C_COMPILER=<C compiler> \
      -DCMAKE_CXX_COMPILER=<C++ compiler> \
      -DOPENMP_TEST_C_COMPILER=<C compiler for testing> \
      -DOPENMP_TEST_CXX_COMPILER=<C++ compiler for testing> \
      -DCMAKE_BUILD_TYPE=Release \
      -DLIBOMP_USE_ARGOBOTS=on \
      -DLIBOMP_ARGOBOTS_INSTALL_DIR=/home/USERNAME/argobots-install \
      |& tee c.txt

for bash and sh:

  cmake ../bolt-VERSION -G "Unix Makefiles" \
      -DCMAKE_INSTALL_PREFIX=/home/USERNAME/bolt-install \
      -DCMAKE_C_COMPILER=<C compiler> \
      -DCMAKE_CXX_COMPILER=<C++ compiler> \
      -DOPENMP_TEST_C_COMPILER=<C compiler for testing> \
      -DOPENMP_TEST_CXX_COMPILER=<C++ compiler for testing> \
      -DCMAKE_BUILD_TYPE=Release \
      -DLIBOMP_USE_ARGOBOTS=on \
      -DLIBOMP_ARGOBOTS_INSTALL_DIR=/home/USERNAME/argobots-install \
      2>&1 | tee c.txt

Bourne-like shells, sh and bash, accept "2>&1 |". Csh-like shell, csh and tcsh, accept "|&". If a failure occurs, the cmake command will display the error. Most errors are straight-forward to follow.

(e) Build BOLT:

for csh and tcsh:

  make |& tee m.txt

for bash and sh:

  make 2>&1 | tee m.txt

This step should succeed if there were no problems with the preceding step. Check file m.txt. If there were problems, do a "make clean" and then run make again with V=1 and VERBOSE=1.

make V=1 VERBOSE=1 |& tee m.txt       (for csh and tcsh)

OR

make V=1 VERBOSE=1 2>&1 | tee m.txt   (for bash and sh)

Then go to step 3 below, for reporting the issue to the BOLT developers and other users.

(f) Install BOLT:

for csh and tcsh:

  make install |& tee mi.txt

for bash and sh:

  make install 2>&1 | tee mi.txt

This step collects all required header and library files in the directory specified by the prefix argument to cmake.

---

2. Testing BOLT ===============

To test BOLT, you can run the test suite. Compilers for testing must be specified when you run cmake.

For example, if llvm-lit is installed:

cd bolt-build
NUM_PARALLEL_TESTS=16
llvm-lit runtime/test -v -j $NUM_PARALLEL_TESTS --timeout 600

If you run into any problems on running the test suite, please follow step 3 below for reporting them to the BOLT developers and other users.

---

3. BOLT-Specific Environmental Variables ===============

BOLT reveals several environmental variables specific to BOLT.

KMP_ABT_NUM_ESS=<int>: Set the number of execution streams which are
                       running on OS-level threads (e.g., Pthreads).
KMP_ABT_SCHED_SLEEP=<1|0>: If it is set to 1, sleep a scheduler when the
                           associate pools are empty.
KMP_ABT_VERBOSE=<1|0>: If it is set to 1, print all the BOLT-specific
                       parameters on runtime initialization.
KMP_ABT_FORK_CUTOFF=<int>: Set the cut-off threshold used for a
                           divide-and-conquer thread creation.
KMP_ABT_FORK_NUM_WAYS=<int>: Set the number of ways for a
                             divide-and-conquer thread creation.
KMP_ABT_SCHED_MIN_SLEEP_NSEC=<int>: Set the minimum scheduler sleep time
                                    (nanoseconds).
KMP_ABT_SCHED_MAX_SLEEP_NSEC=<int>: Set the maximum scheduler sleep time
                                    (nanoseconds).
KMP_ABT_SCHED_EVENT_FREQ=<int>: Set the event-checking frequency of
                                schedulers.
KMP_ABT_WORK_STEAL_FREQ=<int>: Set the random work stealing frequency of
                               schedulers.

---

4. Reporting Problems =====================

If you have problems with the installation or usage of BOLT, please follow these steps:

(a) First visit the Frequently Asked Questions (FAQ) page at https://github.com/pmodels/bolt/wiki/FAQ to see if the problem you are facing has a simple solution.

(b) If you cannot find an answer on the FAQ page, look through previous email threads on the [email protected] mailing list archive (https://lists.bolt-omp.org/mailman/listinfo/discuss). It is likely someone else had a similar problem, which has already been resolved before.

(c) If neither of the above steps work, please send an email to [email protected]. You need to subscribe to this list (https://lists.bolt-omp.org/mailman/listinfo/discuss) before sending an email.

Your email should contain the following files. ONCE AGAIN, PLEASE COMPRESS BEFORE SENDING, AS THE FILES CAN BE LARGE. Note that, depending on which step the build failed, some of the files might not exist.

bolt-build/c.txt (generated in step 1(d) above)
bolt-build/m.txt (generated in step 1(e) above)
bolt-build/mi.txt (generated in step 1(f) above)

DID WE MENTION? DO NOT FORGET TO COMPRESS THESE FILES!

Finally, please include the actual error you are seeing when running the application. If possible, please try to reproduce the error with a smaller application or benchmark and send that along in your bug report.

(d) If you have found a bug in BOLT, we request that you report it at our github issues page (https://github.com/pmodels/bolt/issues). Even if you believe you have found a bug, we recommend you sending an email to [email protected] first.

---

5. Alternate Build Options ==============================

BOLT is based on the OpenMP subproject of LLVM for runtime, and thus it uses the same build options provided in LLVM.

Please visit http://openmp.llvm.org/ for more build options.

bolt's Issues

Disable ittnotify by default in CMake

When building BOLT with Argobots, one needs to set -DLIBOMP_USE_ITT_NOTIFY=off manually, which is tedious. CMake script should handle it (i.e., turn it off by default) when Argobots is specified as a threading layer.

BOLT build fails on ppc64le

This issue was reported in spack/spack#15550

Test failure

Check these tests.
https://jenkins.bolt-omp.org/job/bolt-review-ubuntu/20/
https://jenkins.bolt-omp.org/job/bolt-review-ubuntu/21/

Desire to build bolt with different install directory for omp.h

This simple patch allows the builder of bolt to put omp.h in a different directory. I am working with a more current LLVM omp.h than what bolt builds. With this patch, I can install bolt into the LLVM installation and not clobber omp.h.

git diff
diff --git a/runtime/src/CMakeLists.txt b/runtime/src/CMakeLists.txt
index 042a5200..8a237805 100644
--- a/runtime/src/CMakeLists.txt
+++ b/runtime/src/CMakeLists.txt
@@ -329,7 +329,9 @@ add_dependencies(bolt-libomp-micro-tests bolt-libomp-test-deps)
 # We want to install libomp in DESTDIR/CMAKE_INSTALL_PREFIX/lib
 # We want to install headers in DESTDIR/CMAKE_INSTALL_PREFIX/include
 if(${OPENMP_STANDALONE_BUILD})
-  set(LIBOMP_HEADERS_INSTALL_PATH include)
+  if(NOT LIBOMP_HEADERS_INSTALL_PATH)
+      set(LIBOMP_HEADERS_INSTALL_PATH include)
+  endif()
 else()
   string(REGEX MATCH "[0-9]+\\.[0-9]+(\\.[0-9]+)?" CLANG_VERSION ${PACKAGE_VERSION})
   set(LIBOMP_HEADERS_INSTALL_PATH "${OPENMP_INSTALL_LIBDIR}/clang/${CLANG_VERSION}/include")

FYI, Below are the cmake options I have in our build_bolt.sh script. The above patch allows me to not clobber omp.h. I also turn off the aliases because I want bolt to be user selectable with -fopenmp=libbolt. I have a simple LLVM patch that I want to push upstream to support -fopenmp-libbolt. This does not push the sources for bolt , it simply allows the option to be used when bolt is built as an external component.

MYCMAKEOPTS="
-DCMAKE_INSTALL_PREFIX=$BOLT_INSTALL_DIR
$AOMP_ORIGIN_RPATH
-DCMAKE_C_COMPILER=$AOMP_CC_COMPILER
-DCMAKE_CXX_COMPILER=$AOMP_CXX_COMPILER
-DOPENMP_TEST_C_COMPILER=$AOMP_CC_COMPILER
-DOPENMP_TEST_CXX_COMPILER=$AOMP_CXX_COMPILER
-DCMAKE_BUILD_TYPE=Release
-DOPENMP_ENABLE_LIBOMPTARGET=OFF
-DLIBOMP_HEADERS_INSTALL_PATH=include/bolt
-DLIBOMP_INSTALL_ALIASES=OFF
-DLIBOMP_USE_ARGOBOTS=on"

BOLT seems does not work well with the Dense Linear Lib PLASMA

Hi, I would like to take advantage of lightweight thread libraries, e.g., Argobots, and runtime systems over them, e.g., BOLT. I have successfully configured and built my lib. But I need clarification on the output libs, e.g., libgomp.so, libiomp5.so, libomp.so. Please tell me which one I should use. Do the three libs mean GCC OMP runtime, Intel OMP runtime, and BOLT, respectively？

• Configuration Comand

cmake .. -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=/home/fx/lib/bolt -DCMAKE_C_COMPILER=/home/fx/.local/gcc-11.2.0/bin/gcc -DCMAKE_CXX_COMPILER=/home/fx/.local/gcc-11.2.0/bin/g++ -DOPENMP_TEST_C_COMPILER=/home/fx/.local/gcc-11.2.0/bin/gcc -DOPENMP_TEST_CXX_COMPILER=/home/fx/.local/gcc-11.2.0/bin/g++ -DCMAKE_BUILD_TYPE=Release -DLIBOMP_USE_ARGOBOTS=on -DLIBOMP_ARGOBOTS_INSTALL_DIR=/home/fx/lib/argobots/ | tee c.txt

• Build Comand

make V=1 VERBOSE=1 2>&1 | tee m.txt

• Installed libs
  

run slow when compare to -fopenmp with gcc11

Hi,

I tried to compare bolt + abt with -fopenmp with gcc11, but found it is about 2x slower.
I build abt and bolt according to the guide, both of them use dynamic so.
I wonder if due to this reason.
Is it possible to build bolt as static lib ? And bolt use abt with static lib ?

The test case is matrix mult from taskflow/benchmarks/matrix_multiplication/,
compile command for link bolt as below:
g++ main.cpp omp.cpp taskflow.cpp tbb.cpp -I~/Work/tbb/include -L~/Work/tbb/build/ -ltbb -I~/Work/taskflow -I~/Work/CLI11 -I~/Work/bolt-omp/include -L~/Work/bolt-omp/lib -lbolt -L~/Work/bolt-abt/lib -labt -o test_bolt -O3
./test_bolt -t 2 -m omp

vs
compile command for use default openmp
g++ main.cpp omp.cpp taskflow.cpp tbb.cpp -I~/Work/tbb/include -L~/Work/tbb/build/ -ltbb -I~/Work/taskflow -I~/Work/CLI11 -fopenmp -o test_omp -O3
./test_omp -t 2 -m omp

Hope for some suggestion to get bolt better performance.
Thanks.

inefficient "ordered" scheduling

An ordered loop is not optimized well, which causes sometimes a timeout in test/worksharing/for/omp_for_collapse.c if execution streams are oversubscribed.

  #pragma parallel omp for ordered
  for (i = 1; i < 10000; i++) {
    #pragma omp ordered
    comp(i);
  }

Although order is not widely used, it should be optimized by using synchronization primitives.

To-do list for BOLT 1.0 release

• Upgrade the baseline LLVM/OpenMP to version 10.0
• Upgrade the embedded Argobots to version 1.0
• Support non-x86/64 architectures (including POWER (#48, need further check) and 64-bit ARM)
• Fix reported issues (#46, #49, #51), which will be covered by #69.

BOLT tasks and ABT_cond

I am trying to leverage low-level Argobots features inside BOLT tasks (BOLT 1.0rc3, built with internal Argobots). In particular, I would like to block a set of tasks on a conditional variable and unblock them eventually from a different task, like in this example:

#include <abt.h>
#include <stdio.h>

int main(int argc, char **argv)
{
  int n = 10;
#pragma omp parallel
{
#pragma omp master
{
  int blocked = 0;
  ABT_mutex mtx;
  ABT_cond cond;
  ABT_mutex_create(&mtx);
  ABT_cond_create(&cond);

  for (int i = 0; i < n; ++i) {
    printf("Discovering task %d\n", i);
  #pragma omp task shared(mtx, cond, blocked)
  {
    printf("Task %d blocking\n", i);
    ABT_mutex_lock(mtx);
    blocked++;
    ABT_cond_wait(cond, mtx);
    ABT_mutex_unlock(mtx);
  }
  }

  #pragma omp task shared(cond, mtx, blocked)
  {
    printf("Broadcast task starting\n");
    while (n != blocked) {
      ABT_thread_yield();
    }
    // mutex required to ensure all tasks entered cond
    ABT_mutex_lock(mtx);
    printf("Broadcast task broadcasting\n");
    ABT_cond_broadcast(cond);
    ABT_mutex_unlock(mtx);
  }

  #pragma omp taskwait
}
}
  return 0;
}

What I see is that all tasks are created and only the first task starts executing. Output:

$ ./test_bolt_abt_cond
Discovering task 0
Discovering task 1
Discovering task 2
Discovering task 3
Discovering task 4
Discovering task 5
Discovering task 6
Discovering task 7
Discovering task 8
Discovering task 9
Task 0 blocking

Any idea why only the first task is executing? Are the other runnable tasks not passed to Argobots? Do I need to set some environment variables to make this work?

Child task management

The child task management design needs to be improved.

The first implementation is done by checking the queued tasks from the current task to the end of the queue and free those tasks so we can assert that children tasks are completed before the parent one continues.

The future idea is to manage children tasks inside the task structure so each task will be able to check them directly without involving the thread structure

Occasional failure in the omp_for_schedule_guided test

worksharing/for/omp_for_schedule_guided.c fails sometimes.
https://jenkins-pmrs.cels.anl.gov/job/bolt-review-centos/78/

This should be fixed.

Parallel Compilation Error

When I try to use an embedded Argobots library(external/argobots) and compile BOLT in parallel (e.g., make -j), it causes the following compile-time error:

$ cmake -DLIBOMP_USE_ARGOBOTS=on
...
$ make -j
Performing configure step for 'libabt'
...
bolt/runtime/src/kmp.h:23:17: fatal error: abt.h: No such file or directory
 #include <abt.h>

Add an address sanitizer option for BOLT CI

Argobots supports an address sanitizer (pmodels/argobots#218) and seems working with relatively new GCC and Clang (https://jenkins-pmrs.cels.anl.gov/view/abt/job/argobots-review-centos-all/). Since tricky user-level context switching in Argobots passes the tests, BOLT should also pass those address sanitizers. The BOLT CI should use address sanitizers.

Testsuite with Fortran and Intel compilers

Build information:

• Compiler: Intel 17.2.050
• BOLT build: c.txt m.txt mi.txt

Testsuite run parameters:

• TEST_FC=ifort
• TEST_FFLAGS="-g -O2 -qopenmp -I/~/INSTALL/bolt/include -L/~/INSTALL/bolt/lib -Wl,-rpath=~/INSTALL/bolt/lib"`

make ftest error message:

Testing for "omp_threadprivate":
Generating sources .............. success
Compiling soures ................ success
Running test with 8 threads ../bin/fortran/test_omp_threadprivate: relocation error: ./bin/fortran/test_omp_threadprivate: symbol kmp_aligned_malloc, version VERSION not defined in file libiomp5.so with link time reference
.... failed 100% of the tests

Failure configuring with internal libabt

Trying to build BOLT with internal Argobots on current master results in a CMake error:

$ cmake ../ -DCMAKE_INSTALL_PREFIX=$HOME/opt/bolt-git -DLIBOMP_USE_ARGOBOTS=on
...
CMake Error at /usr/share/cmake-3.10/Modules/ExternalProject.cmake:2474 (message):
  No download info given for 'libabt' and its source directory:

   ~/src/bolt/git/bolt/external/argobots

  is not an existing non-empty directory.  Please specify one of:

   * SOURCE_DIR with an existing non-empty directory
   * DOWNLOAD_COMMAND
   * URL
   * GIT_REPOSITORY
   * SVN_REPOSITORY
   * HG_REPOSITORY
   * CVS_REPOSITORY and CVS_MODULE
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/ExternalProject.cmake:3029 (_ep_add_download_command)
  external/CMakeLists.txt:32 (ExternalProject_Add)

Add `#include <stdlib.h>` to examples

I get this warning when I try to compile any of the examples.

 warning: incompatible implicit declaration of built-in function 'malloc'
     double *a = (double *)malloc(sizeof(double)*num);

I was able to fix it by adding #include <stdlib.h> to the examples

Are task dependencies (depend clauses) in bolt working?

This could be my user error. I have a simple code where I am trying to get T2 to execute after T1 because T1 writes x which is input to T2. The code is not setting x to 2 before T2 runs. Here is the code. This fails the same way in both libgomp and libbolt. It fails in clang and gcc 7.5.

#include <stdio.h>
#include <omp.h>
int foo() {
   int x = 1;
   int x_is_not_equal_two=0;
   #pragma omp task depend(in:x) shared(x_is_not_equal_two,x)
   {
      if (x != 2) {
         x_is_not_equal_two = 1;
         printf(" T2: INPUT dependend clause x should now be 2  x:%d\n",x);
      }
   }
   #pragma omp task depend(out:x) shared(x)
   {
      printf(" T1: OUTPUT setting x to 2\n");
      x=2;
   }
   printf("before taskwait x:%d\n",x);
   #pragma omp taskwait
   printf("after  taskwait x:%d  x  was not equal 2 in T2:%d \n", 
      x, x_is_not_equal_two);
   return x_is_not_equal_two;
}

int main() {
   int rc=0;
   omp_set_num_threads(2);
   #pragma omp parallel
   #pragma omp single nowait
   rc = foo();
   printf("rc:%d\n",rc);
   return rc;
}

Segfault in ABT_mutex_lock

Environment:

• LLVM 3.9.1
• BOLT 1.0a1
• Argobots 1.0a1

Reproducible code:
bolt_abt_mtx_segf.txt

Error message:

[mutex.c:172] ABT_mutex_lock: 20
Segmentation fault (core dumped)

GDB output:

Program terminated with signal 11, Segmentation fault.
#0  0x00007f64ca2426c2 in ABT_mutex_create ()
   from /usr/local/llvm-bolt-argobots/lib/../lib/libabt.so.0
 (gdb) where
 #0  0x00007f64ca2426c2 in ABT_mutex_create ()
   from /usr/local/llvm-bolt-argobots/lib/../lib/libabt.so.0
#1  0x00007f64caa75300 in __kmp_do_serial_initialize ()
    at /usr/local/src/llvm/projects/openmp/runtime/src/abt/kmp_abt_runtime.c:139
#2  0x00007f64caa75a5a in __kmp_do_middle_initialize ()
    at /usr/local/src/llvm/projects/openmp/runtime/src/abt/kmp_abt_runtime.c:363
#3  0x00007f64caa75ddb in __kmp_middle_initialize ()
    at /usr/local/src/llvm/projects/openmp/runtime/src/abt/kmp_abt_runtime.c:470
#4  0x00007f64caa99362 in __kmp_api_omp_get_num_procs ()
    at /usr/local/src/llvm/projects/openmp/runtime/src/abt/kmp_abt_ftn_entry.h:444
#5  0x0000000000400b09 in main ()```

thread id inside omp task code management

Now, the thread information for each omp task is taken from blocked ULTs. It works from common task patterns but not for nested task codes that start with single or master clauses (like Fibonacci).

The problem is that we are doubling the thread structures including the task queue so when a new omp task is created it can be done in the real threads queue or in the "fake" one. Then, when we try to join the task... we can not be sure where it is.

However, this issue is related to the OpenMP committee decision about if two omp task can be concurrently executed sharing the thread id

Untied tasks trigger assert

Compiling the following code snippet with clang 9 and running it with BOLT (1.0rc3 downloaded from website, built with Argobots support) leads to an assertion:

int main(int argc, char **argv)
{
#pragma omp parallel
#pragma omp master
{
  #pragma omp task untied
  { }
}
  return 0;
}

The assertion and backtrace:

[New Thread 0x7ffff6d1f700 (LWP 19430)]
[New Thread 0x7ffff651e700 (LWP 19431)]
[New Thread 0x7ffff5d1d700 (LWP 19432)]
Assertion failure at z_Linux_util.cpp(3952): taskdata->td_flags.complete == 0.
OMP: Error #13: Assertion failure at z_Linux_util.cpp(3952).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.

Thread 3 "test_omp_nested" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff651e700 (LWP 19431)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff751f801 in __GI_abort () at abort.c:79
#2  0x00007ffff7b3186a in __kmp_abort_process () at bolt-1.0rc3/runtime/src/kmp_runtime.cpp:463
#3  0x00007ffff7b2f765 in __kmp_fatal (message=...) at bolt-1.0rc3/runtime/src/kmp_i18n.cpp:868
#4  0x00007ffff7b2befd in __kmp_debug_assert (msg=0x7ffff7bb0ada "taskdata->td_flags.complete == 0", file=0x7ffff7bb3c37 "z_Linux_util.cpp", line=3952)
    at bolt-1.0rc3/runtime/src/kmp_debug.cpp:74
#5  0x00007ffff7b8dec2 in __kmp_abt_execute_task (arg=0xb1ee40) at bolt-1.0rc3/runtime/src/z_Linux_util.cpp:3952
#6  0x00007ffff72d1beb in ABTD_thread_func_wrapper_thread (p_arg=0x7fffe6802f00) at ../../src/arch/abtd_thread.c:18
#7  0x00007ffff72d1fb1 in make_fcontext () at ../../src/arch/fcontext/make_x86_64_sysv_elf_gas.S:64
#8  0x00007fffe6802e00 in ?? ()
#9  0x00007ffff7b8de00 in ?? () at bolt-1.0rc3/runtime/src/z_Linux_util.cpp:3441 from ~/opt/bolt-1.0rc3/lib/libomp.so.5
#10 0x0000000000a13f80 in ?? ()
#11 0x00007fffe6fff000 in ?? ()
#12 0x0000000000000000 in ?? ()

I don't see this happening with tied tasks.