gerazo / pipert

Real-Time meets High-Delay: A hybrid scheduling and dataflow framework for DSP applications

PipeRT

Real-Time meets High-Delay: A hybrid scheduling and dataflow framework for DSP applications, developed at ELTE University, Budapest.

Features

High performance:

• Multithreaded scheduler with worker pool
• In-place running in memory, allocations are only made during the setup phase, none are made during the running phase
• Uses mostly non-blocking algorithms avoiding OS locks

Easy to use:

• Users don't need to have mutexes or any kind of synchronization primitives
• Nice, compact OOP API with templates to support custom data types
• Cross-platform C++11 code tested on gcc and clang

Versatile:

• Configurable buffers to support low- and high delay units
• Supports stateless and stateful (accumulate) tasks
• Supports user controlled threads besides own thread pool to support UI, I/O or special thread affinity
• Supports circles (feedback) in the pipeline chain
• Supports multiple entry and exit (even reentry) points for distributed workflows

Measurable (coming soon):

• Supports measurement-driven development via its own, minimal impact autoprofiling system
• Has a measurement API which exposes autoprofiling capabilities of the whole pipeline
• Pluggable measurements: The user can fit any logging mechanism to the profiler

Overview

The scheduler manages a pool of worker threads to complete work in user-defined nodes on user-defined processing functions. Initially, when the scheduler is instantiated, it enters the setup phase. In this phase, channels should be created that represent the data processing stages in the pipeline using packet processing functions defined in the user's nodes. There can be two types of channels, scheduled or polled. The former channel type's packets are automatically scheduled to be processed by the worker pool, and the latter type's packets can be received manually be the user's own threads. This can be useful when the use case requires a dedicated thread for certain actions, such as when you need to process data using non-thread-safe librares, or when you need to make an exit point from the DSP pipeline to transmit some results out. When all channels are created, the scheduler can be started, at which point packets can start entering the pipeline.

The processing functions defined in the user's nodes accept a packet to be processed as a parameter. After they are done processing a packet, packets can be sent onto the next channel (one that the user node has knowledge of) in the pipeline by acquiring a packet to be filled from that channel and filling it with the processed data. The filled packet is automatically queued for processing in the next channel upon the destruction of the stub belonging to the filled packet. The space in memory belonging to the processed packet is freed up for reuse automatically upon the destruction of the stub belonging to the packet.

At any point after the scheduler was started, it may be stopped, which means it will instruct all worker threads to finish their processing, wait for them to complete their current tasks, and re-enter the setup/preparation phase.

For a more detailed description on how each component in the system works, please check out our documentation.

Normal Build for Your Project

You need the following packages installed on your system to build the library:

• git for downloading this repo
• cmake and ninja-build to make and use the build system
• gcc, g++ for compiling with GCC OR
• clang, clang++ and llvm for compiling with Clang
• [optional] doxygen to generate HTML documentation

Steps to build:

1. Download the repo using git clone https://github.com/gerazo/pipert.git
2. Run ./build.sh to build the static and dynamic library in release mode
3. What you need for your project:
   • the header files in our include folder
   • the dynamic library, build/libpipert.so OR
   • the static library, build/libpipert_static.a

You can find the documentation under build/docs/index.html.

Development Build for Contributors

Besides everything under Normal Build, you may also want the following packages:

• clang-format to keep our code formatting conventions
• gcovr to generate text and HTML coverage reports

Steps for development build:

1. Checkout the repo
2. Run ./devbuild.sh or choose a generator for your IDE by using ./devbuild.sh -G "<yourIDE> - Ninja" (see cmake -h)
3. Find the debug build under build_debug and the release build under build_release folders
4. In any of these, open your generated IDE project or use ninja and ninja test targets from the command line
5. Coverage reports will only be generated for the debug build into build_debug/coverage/coverage.html and can be regenerated by issuing ninja coverage at the command line
6. Documentation will only be generated for the release build by default into build_release/docs/index.html, but generation can be forced by issuing ninja docs at the command line

To only build in debug or release mode, you should specify both the build directory and the build mode by running ./devbuild.sh -dir <build_debug|build_release> -mode <Debug|Release>.

Usage examples

Soon, we will have some tests that also show simple examples of usage.

Support for different platforms

The project is currently developed on *nix machines. Development on Windows or OS X systems is not supported as of yet, but execution on Windows machines is currently being looked into.

Have fun!

pipert's Issues

Test cases for ScheduledChannel

1st version of Documentation

Finish docs branch with

• new comments on recently introduced functions (#9)
• putting images in it

Dropping packets issue with acquiring before push

Dropping packets automatically behaves unexpected when the available Packets are Acquired already, but haven't been Pushed yet.

For details check: #8 test/PolledChannelTest.cpp file PolledChannelDropPacketBeforePushTest test case.

Reason:
The freepackets is already empty, however, because the pushing of the packets haven't happened yet, the queuedpackets is also empty, so the Release method will return back with a nullptr when it tries to drop and free packet.

Pushed event from N/A channels as sender when the recieving channels is not an input one

Graph for this issue:

Packets demostrate this issue:

Receiver: PrinterChannel
Sender: N/A
Events: [
type: Execution Time, log count: 100, time passed: 1663871, min: 0.0, max: 35975.0, avg: 16635.23
,
type: Packet Retrieved, log count: 100, time passed: 1663871, min: 32.0, max: 1663629.0, avg: 1159050.84
,
type: Read Time, log count: 100, time passed: 1663871, min: 0.0, max: 35975.0, avg: 16635.05
,
type: Fill Time, log count: 100, time passed: 1663871, min: 0.0, max: 2.0, avg: 0.12
,
type: Packet Pushed, log count: 100, time passed: 1663871, min: 3.0, max: 60.0, avg: 34.74
]
Receiver: sqrtChannel
Sender: N/A
Events: [
type: Packet Dropped, log count: 1, time passed: 10611, min: 1.0, max: 1.0, avg: 1.0
,
type: Execution Time, log count: 3, time passed: 10611, min: 2.0, max: 3.0, avg: 2.6666666666666665
,
type: Packet Retrieved, log count: 3, time passed: 10610, min: 46282.0, max: 83822.0, avg: 67350.66666666667
,
type: Read Time, log count: 3, time passed: 10611, min: 2.0, max: 3.0, avg: 2.6666666666666665
,
type: Fill Time, log count: 4, time passed: 10611, min: 0.0, max: 2.2250738585072014e-308, avg: 0.0
,
type: Packet Pushed, log count: 4, time passed: 10611, min: 6222.0, max: 83819.0, avg: 52065.0
]
Receiver: OutChannel
Sender: N/A
Events: [
type: Packet Pushed, log count: 1, time passed: 3026, min: 136195.0, max: 136195.0, avg: 136195.0
]

Incorrect profiled coverage reports after incremental builds

Incrementally building the project and not deleting the .gcda files generated by gcov seems to yield incorrect execution profiling information. Every time tests are re-run, all object files that are used by the test will have an associated .gcda file generated for them. These .gcda files should probably be replaced upon re-running the test, instead of whatever gcov is doing right now.

A possible solution would be to delete all .gcda files associated with a test's execution before executing the test again and generating runtime coverage information from it.

Build artifacts upload

We could upload the generated coverage information and documentation to GitHub Actions artifacts, so we could read the docs and coverage online after every build. In this case, we could even have separate the single "build" job with multiple steps into more jobs with less steps. For instance, we could have

• gcc job with debug and release passes uploading doc and coverage artifacts
• clang job with debug and release
• win job with debug and release

Better ReadMe for beginners

We should help beginners who may not know how to

• check a repository out
• install packages on their system
• compile this library into their own application

Use anonymous namespace in every test

Check that every of our tests use the anonymous namespace for local definitions!

Separate heap-in-fixed-size-vector container functionality into separate class

We are using a special type of heap in ChannelImpl::queued_packets_, SchedulerImpl::state_queue_ and in SchedulerImpl::state2channel_queues_ as the value side of the key.value pairs. These heaps have the following important features:

• They are std::vectors which do not change during runtime. They are constant in size during operation.
• They are heaps inside and use std::push_heap() and std::pop_heap() algorithms to keep up the heap condition.
• They accept custom ordering (see the code) between the elements (but this should be a strict ordering).
• They have a special operation: they can update an element in the heap by removing and re-adding it. (See the code!)

It would be good to have such a class separately in order to keep the two mentioned implementation classes minimal.
This is a rather complex refactoring task.

PolledChannel class unit testing

Hotfix: MergeTest is flaky on Clang debug

StateScheduling.MergeSchedulingTest fails occasionally but only in Clang debug build (Clang 7 and 10 are also affected).

[ RUN      ] StateScheduling.MergeSchedulingTest
../test/StateSchedulingTest.cpp:203: Failure
Expected equality of these values:
  merger.GetValue() + dropped_packets1 - dropped_packets2
    Which is: 1
  0
[  FAILED  ] StateScheduling.MergeSchedulingTest (17 ms)

Scheduler immediate start-stop deadlock

Could someone please review this change and confirm that this analysis is correct?

Recently, I've implemented a simple test that starts and stops the scheduler twice.

TEST(Scheduler, StartStopRestartTest) {
  pipert::Scheduler sch;

  EXPECT_FALSE(sch.IsRunning());
  sch.Start();
  EXPECT_TRUE(sch.IsRunning());
  sch.Stop();
  EXPECT_FALSE(sch.IsRunning());
  sch.Start();
  EXPECT_TRUE(sch.IsRunning());
  sch.Stop();
  EXPECT_FALSE(sch.IsRunning());
}

This test deadlocked the pipert_bbtest found in build_debug about every ten runs or so. After hitting re-runs in gdb for a while, I got the following backtrace:

GDB log

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff45ff700 (LWP 10867)]
[New Thread 0x7ffff3dfe700 (LWP 10868)]
[New Thread 0x7ffff35fd700 (LWP 10869)]
[New Thread 0x7ffff2dfc700 (LWP 10870)]
[Thread 0x7ffff2dfc700 (LWP 10870) exited]
[Thread 0x7ffff35fd700 (LWP 10869) exited]
[Thread 0x7ffff3dfe700 (LWP 10868) exited]
[Thread 0x7ffff45ff700 (LWP 10867) exited]
[New Thread 0x7ffff2dfc700 (LWP 10871)]
[New Thread 0x7ffff35fd700 (LWP 10872)]
[New Thread 0x7ffff3dfe700 (LWP 10873)]
[New Thread 0x7ffff45ff700 (LWP 10874)]
[Thread 0x7ffff35fd700 (LWP 10872) exited]
[Thread 0x7ffff2dfc700 (LWP 10871) exited]
[Thread 0x7ffff3dfe700 (LWP 10873) exited]

Thread 1 "pipert_bbtest" received signal SIGINT, Interrupt.
0x00007ffff7e8f495 in __GI___pthread_timedjoin_ex (threadid=140737293317888, thread_return=0x0, abstime=0x0, 
  block=<optimized out>) at pthread_join_common.c:89
89	pthread_join_common.c: No such file or directory.

Thread 9 (Thread 0x7ffff45ff700 (LWP 10874)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x611000002608)
  at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x606000000ff0, cond=0x6110000025e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x6110000025e0, mutex=0x606000000ff0) at pthread_cond_wait.c:655
#3  0x00007ffff7db83bc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
 from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff7f3108d in std::_V2::condition_variable_any::wait<std::unique_lock<pipert::AdaptiveSpinLock> > (
  this=0x6110000025e0, __lock=...)
  at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/condition_variable:263
#5  0x00007ffff7f1e17a in pipert::SchedulerImpl::RunTasks (this=0x6110000025c0) at ../src/SchedulerImpl.cpp:174
#6  0x00007ffff7f46ec1 in std::__invoke_impl<void, void (pipert::SchedulerImpl::*)(), pipert::SchedulerImpl*> (
  __f=@0x603000004a00: (void (pipert::SchedulerImpl::*)(class pipert::SchedulerImpl * const)) 0x7ffff7f1dc90 <pipert::SchedulerImpl::RunTasks()>, __t=@0x6030000049f8: 0x6110000025c0)
  at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/invoke.h:73
#7  0x00007ffff7f46c06 in std::__invoke<void (pipert::SchedulerImpl::*)(), pipert::SchedulerImpl*> (
  __fn=@0x603000004a00: (void (pipert::SchedulerImpl::*)(class pipert::SchedulerImpl * const)) 0x7ffff7f1dc90 <pipert::SchedulerImpl::RunTasks()>, __args=@0x6030000049f8: 0x6110000025c0)
  at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/invoke.h:95
#8  0x00007ffff7f46b91 in std::thread::_Invoker<std::tuple<void (pipert::SchedulerImpl::*)(), pipert::SchedulerImpl*> >::_M_invoke<0ul, 1ul> (this=0x6030000049f8)
  at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/thread:244
#9  0x00007ffff7f46b11 in std::thread::_Invoker<std::tuple<void (pipert::SchedulerImpl::*)(), pipert::SchedulerImpl*> >::operator() (this=0x6030000049f8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/thread:253
#10 0x00007ffff7f46588 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (pipert::SchedulerImpl::*)(), pipert::SchedulerImpl*> > >::_M_run (this=0x6030000049f0)
  at /usr/bin/../lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/thread:196
#11 0x00007ffff7dbdb2f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff7e8dfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#13 0x00007ffff7a8c4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7991cc0 (LWP 10866)):
#0  0x00007ffff7e8f495 in __GI___pthread_timedjoin_ex (threadid=140737293317888, thread_return=0x0, abstime=0x0, 
  block=<optimized out>) at pthread_join_common.c:89
#1  0x00007ffff7dbdd53 in std::thread::join() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007ffff7f1cc20 in pipert::SchedulerImpl::Stop (this=0x6110000025c0) at ../src/SchedulerImpl.cpp:65
#3  0x00007ffff7f1b449 in pipert::Scheduler::Stop (this=0x7fffffffd680) at ../src/Scheduler.cpp:28
#4  0x000000000054b3cb in Scheduler_StartStopRestartTest_Test::TestBody (this=0x6020000012f0)
  at ../test/SchedulerTest.cpp:53
#5  0x00000000005db51e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (
  object=0x6020000012f0, method=&virtual testing::Test::TestBody(), location=0x6e2135 "the test body")
  at googletest-src/googletest/src/gtest.cc:2433
#6  0x00000000005c3f0b in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (
  object=0x6020000012f0, method=&virtual testing::Test::TestBody(), location=0x6e2135 "the test body")
  at googletest-src/googletest/src/gtest.cc:2469
#7  0x00000000005a40c9 in testing::Test::Run (this=0x6020000012f0) at googletest-src/googletest/src/gtest.cc:2508
#8  0x00000000005a4e4b in testing::TestInfo::Run (this=0x612000001b40) at googletest-src/googletest/src/gtest.cc:2684
#9  0x00000000005a5567 in testing::TestSuite::Run (this=0x6120000019c0) at googletest-src/googletest/src/gtest.cc:2816
#10 0x00000000005b26e5 in testing::internal::UnitTestImpl::RunAllTests (this=0x615000000080)
  at googletest-src/googletest/src/gtest.cc:5338
#11 0x00000000005df19e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x615000000080, 
  method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x5b2270 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x6e294d "auxiliary test code (environments or event listeners)")
  at googletest-src/googletest/src/gtest.cc:2433
#12 0x00000000005c674b in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x615000000080, 
  method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x5b2270 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x6e294d "auxiliary test code (environments or event listeners)")
  at googletest-src/googletest/src/gtest.cc:2469
#13 0x00000000005b2216 in testing::UnitTest::Run (this=0x10bd528 <testing::UnitTest::GetInstance()::instance>)
  at googletest-src/googletest/src/gtest.cc:4925
#14 0x00000000005994f1 in RUN_ALL_TESTS () at googletest-src/googletest/include/gtest/gtest.h:2473
#15 0x00000000005994d6 in main (argc=1, argv=0x7fffffffe4e8) at googletest-src/googletest/src/gtest_main.cc:45

From this log we can see that the main thread who spawned the workers is infinitely waiting for a worker thread to join it. A single thread in this case is still waiting. More specifically, thread 9 is waiting for global_covar_ in line 174 of SchedulerImpl.cpp.

My theory for why this could happen: thread 9 has started executing later than the other threads, and the main thread in the test is till running, and it already entered SchedulerImpl::Stop(). Thread 9 has acquired the lock on global_mutex_ and saw that keep_running_ was still true. After this, the main thread writes false into keep_running and notifies all threads who are waiting on global_covar_ to wake up. However, thread 9 still hasn't started waiting for global_covar_ so it misses the notify_all() issued by the main thread. after this, thread 9 starts waiting for global_covar_, and the main thread tries to wait out the termination of all threads using join().

It seems to me that the problem is that worker threads use a lock on global_mutex_ before checking the value of keep_running_, but the main thread modifies the latter without any synchronization. I tried to fix this simply by guarding the modification of it using the same mutex. See the following diff.

diff

diff --git a/src/SchedulerImpl.cpp b/src/SchedulerImpl.cpp
index 737f0bd..e4eebef 100644
--- a/src/SchedulerImpl.cpp
+++ b/src/SchedulerImpl.cpp
@@ -59,7 +59,10 @@ void SchedulerImpl::Stop() {
   assert(running_.load(std::memory_order_acquire));
   if(!running_.load(std::memory_order_acquire))
     return;
-  keep_running_.store(false, std::memory_order_release);
+  {
+    std::unique_lock<AdaptiveSpinLock> lock(global_mutex_);
+    keep_running_.store(false, std::memory_order_release);
+  }
   global_covar_.notify_all();
   for (auto& t : workers_) {
     t.join();

And sure enough, after slamming gdb with at least a hundred re-runs of the test has never deadlocked since.

The code can be found in the branch unit_testing. The issue can be reproduced in 49e60af and was (probably) fixed in 5e97695.

Log Analyzer and Performance Monitor

We need an app which will be able to

• Receive logs created by #6
• Show logs in real-time and visualize them
• Make certain analysis
  • Weakest points, bottlenecks
  • Overall system performance and latency
  • Dataflow mapping, pipeline visualizatun of current system

Thread identification utility

We need a small utility which can identify whether the running thread is

• ours from our worker pool
• custom user thread we have never seen before
• custom user thread we have already tagged

This is needed to be able to log correctly which job units are executed by which resource.
This is a requirement for #6

Scheduled channel start error

Test code

TEST(SchedulerTest, ChannelCreation) {
    pipert::Scheduler sch(1);

    Printer p;
    pipert::ScheduledChannel<Human> sc = sch.CreateScheduledChannel<Human>("ScheduledChannel", 2, nullptr, std::bind(&Printer::Print, &p, std::placeholders::_1));

    EXPECT_EQ(sc.GetCapacity(), 2);
    EXPECT_EQ(sc.GetPacketSize(), sizeof(pipert::Packet<Human>));
    EXPECT_EQ(sc.GetName(), "ScheduledChannel");

    sch.Start();
}

Test log
pipert_bbtest: ../src/SchedulerImpl.cpp:39: void pipert::SchedulerImpl::UnregisterChannel(pipert::ChannelImpl*): Assertion `!running_.load(std::memory_order_acquire)' failed.

Less significant problem: Scheduler

if the scheduler started already and then I call function start again, it gives an error.

sch1.Start();
sch1.Start();
EXPECT_EQ(sch1.IsRunning(), true);
// Assertion `!running_.load(std::memory_order_acquire)' failed.

Checking Windows compatibility

We should check how it is possible to compile, link and possibly develop the library on Windows. Question?

• Which emulation or compatibility layer is suggested for running the build?
• What extra steps are needed?
• How to compile and link to an application?
• Which compilers are supported?
• How to install the requirements (with links possibly).

The main goal is to support native compilation and linking to an other project, so native usage is the priority, not native development. (it is ok to require a compatibility layer from devs.)

Protocol plugin to add remote resources to the pipeline

Given the latest version of the scheduler.
When I want to involve remote computers into the pipeline.
Then I need to handle this functionality in my code.
And I would like a plugin to solve network communication in the public API.

This feature would be helpful to use dedicated computers in the pipeline for a given task (e.g. a computer with proper GPU for object detection/ recognition).

cmake build for demos

I get the following error when I try to build and run demos :
I use the command cmake --build for building source files there , is there any additional setting I should do before build?
"REAL_BBTEST_SOURCES

/usr/bin/ld: cannot find -lpipert
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/simple.dir/build.make:84: simple] Error 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/simple.dir/all] Error 2
make: *** [Makefile:84: all] Error 2 "

wrong detection for the build in run.sh

The build is detected through the existence of the directory, even though the directory can exist by a failed build.

Buffer change on Clang release only

See #51 for details!

Measurement Module Concept (code and architectural change)

check needed change in system in order to add new classes which hold user setting for custom define measurement to be measured in the code , runtime workflow to process those measurement and output generation format