circonus-labs / libmtev

Mount Everest Application Framework

libmtev - a toolkit for building high-performance servers.

Read the documentation.

To build, check out the instructions in BUILDING.

To run the test suite, run make check from the test/ directory.

Enjoy.

An incomplete list of features.

• process manager/watchdog/monitor/crash catcher
• powerful config system with write-back on changes
• eventing system for sockets, timers and asynch jobs
  • Multi-Threaded EVent loops - (mtev)
  • dynamically scaling asynchronous jobqs
• module system
• network listener system (w/ TLS)
  • REST/http routing convenience layer
  • channelized reverse tunneling of TCP.
• telnet-accessible command-control system
• epoch memory reclamation
• clustering subsystem
• robust rest-accessible statistics (w/ histograms)
• fq and amqp connectors
• DTrace (and Systemtap/eBPF) USDT probes
• opentracing (zipkin thrift) support
• mdb helpers (Illumos)
• various data structures
• accelerated timing support (faster than OS)
• dynamic hooks and runtime resolveable callsites
• embedded luajit w/ stand-alone lua runtime
• mtevbusted (mtev capable lua busted testing suite)

TODO

libmtev's Issues

Support multi-certificate TLS via SNI

Add instrumentation around lua memory/GC

Whishlist:

• lua_gc latency
• size of managed memory
• count of table allocations

Logo design

Hello there. I am a graphic designer. I want to design a logo for the readme. If you like it. I will give you present.

Change default jobq memory safety to "gc"

This is a one-line change, but a breaking one for apps. Luckily, today, almost no one is using SMR. Do a quick audit of apps using SMR first.

Is this project related to CIrconus-IRONdb?

I was learning document of IRONdb and found out that release note links to this repository.
I have many questions about IRONdb, is it supposed to disccuss here?

issue compiling on debian jessie

After remove a test for lua.h (which seems to fail on jessie), I was able to get configure to run.

thomas@circonus:~/libmtev$ git diff
diff --git a/configure.in b/configure.in
index 82f6788..856dd37 100755
--- a/configure.in
+++ b/configure.in
@@ -238,7 +238,7 @@ if test "x$ENABLE_LUA" != "xno"; then
                AC_MSG_ERROR([*** liblua52 required ***])
        fi

-       AC_CHECK_HEADER(lua.h, [], [AC_MSG_ERROR([*** lua (headers) required ***])])
+#      AC_CHECK_HEADER(lua.h, [], [AC_MSG_ERROR([*** lua (headers) required ***])])
        AC_CHECK_HEADER(udns.h, [], [AC_MSG_ERROR([*** udns.h required ***])])
        AC_CHECK_LIB(udns, dns_new, [], [AC_MSG_ERROR([*** libudns required ***])])
 fi

I still expect an issue when it comes to handling lua (but hopefully it will be not too hard to hand edit the Makefile to fix the include and lib paths).

But more importantly It seems that some API changed with libck

make[2]: Entering directory '/home/thomas/libmtev/src/utils'
mtev_memory.c: In function ‘ck_epoch_barrier_true’:
mtev_memory.c:60:20: warning: passing argument 1 of ‘ck_epoch_barrier’ from incompatible pointer type
   ck_epoch_barrier(r);
                    ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:159:6: note: expected ‘struct ck_epoch_t *’ but argument is of type ‘struct ck_epoch_record_t *’
 void ck_epoch_barrier(ck_epoch_t *, ck_epoch_record_t *);
      ^
mtev_memory.c:60:3: error: too few arguments to function ‘ck_epoch_barrier’
   ck_epoch_barrier(r);
   ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:159:6: note: declared here
 void ck_epoch_barrier(ck_epoch_t *, ck_epoch_record_t *);
      ^
mtev_memory.c: In function ‘mtev_memory_barriers’:
mtev_memory.c:67:21: warning: assignment from incompatible pointer type
     else do_cleanup = ck_epoch_poll;
                     ^
mtev_memory.c: In function ‘mtev_memory_maintenance’:
mtev_memory.c:74:37: warning: assignment from incompatible pointer type
   if(do_cleanup == NULL) do_cleanup = ck_epoch_poll;
                                     ^
mtev_memory.c: In function ‘mtev_memory_begin’:
mtev_memory.c:90:18: warning: passing argument 1 of ‘ck_epoch_begin’ from incompatible pointer type
   ck_epoch_begin(epoch_rec, NULL);
                  ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:87:1: note: expected ‘struct ck_epoch_t *’ but argument is of type ‘struct ck_epoch_record_t *’
 ck_epoch_begin(ck_epoch_t *epoch, ck_epoch_record_t *record)
 ^
mtev_memory.c: In function ‘mtev_memory_end’:
mtev_memory.c:93:16: warning: passing argument 1 of ‘ck_epoch_end’ from incompatible pointer type
   ck_epoch_end(epoch_rec, NULL);
                ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:123:1: note: expected ‘struct ck_epoch_t *’ but argument is of type ‘struct ck_epoch_record_t *’
 ck_epoch_end(ck_epoch_t *global, ck_epoch_record_t *record)
 ^
mtev_memory.c: In function ‘mtev_memory_ck_free_func’:
mtev_memory.c:158:19: warning: passing argument 1 of ‘ck_epoch_call’ from incompatible pointer type
     ck_epoch_call(epoch_rec, &e->epoch_entry, f);
                   ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:139:1: note: expected ‘struct ck_epoch_t *’ but argument is of type ‘struct ck_epoch_record_t *’
 ck_epoch_call(ck_epoch_t *epoch,
 ^
mtev_memory.c:158:30: warning: passing argument 2 of ‘ck_epoch_call’ from incompatible pointer type
     ck_epoch_call(epoch_rec, &e->epoch_entry, f);
                              ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:139:1: note: expected ‘struct ck_epoch_record_t *’ but argument is of type ‘struct ck_epoch_entry_t *’
 ck_epoch_call(ck_epoch_t *epoch,
 ^
mtev_memory.c:158:47: warning: passing argument 3 of ‘ck_epoch_call’ from incompatible pointer type
     ck_epoch_call(epoch_rec, &e->epoch_entry, f);
                                               ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:139:1: note: expected ‘struct ck_epoch_entry_t *’ but argument is of type ‘void (*)(struct ck_epoch_entry_t *)’
 ck_epoch_call(ck_epoch_t *epoch,
 ^
mtev_memory.c:158:5: error: too few arguments to function ‘ck_epoch_call’
     ck_epoch_call(epoch_rec, &e->epoch_entry, f);
     ^
In file included from mtev_memory.c:34:0:
/usr/include/ck_epoch.h:139:1: note: declared here
 ck_epoch_call(ck_epoch_t *epoch,
 ^
Makefile:36: recipe for target 'mtev_memory.lo' failed

The version installed is

$ dpkg -l | grep libck
ii  libck-dev                      0.3.5-1                     amd64        Concurrency Kit - development files
ii  libck0                         0.3.5-1                     amd64        Concurrency Kit - shared libraries

$ cat /etc/debian_version
8.3

mtev_json_object ref counting should be atomic

extern struct mtev_json_object* mtev_json_object_get(struct mtev_json_object *jso)
{
  if(jso) {
    jso->_ref_count++;
  }
  return jso;
}
extern void mtev_json_object_put(struct mtev_json_object *jso)
{
  if(jso) {
    jso->_ref_count--;
    if(!jso->_ref_count) jso->_delete(jso);
  }
}

Object shared among many threads, get/put pattern results in a bad ref count and either a leak or a crash when jso is freed while there are still references to it.

Ref counting should probably be atomic (ck_pr_inc/dec_64 or similar)

Allow divesting of mtev_mm_barrier_asynch responsibilities

When threads call mtev_memory_maintenance_ex(MTEV_MM_BARRIER_ASYNCH), it delays work that they will do on some subsequent call to the same function. However, when threads shut down, they must wait for all outstanding reclamation requests to complete before the thread will terminate (mtev_memory_fini_thread). This is highly undesirable in the job queue thread wind-down scenario and can cause starvation and backlogs.

There should be an option for threads to "disown" reclamation responsibilities and this should called during shutdown to allow a prompt return from mtev_memory_fini_thread.

badly referenced dependencies: jlog

Hello,

> autoconf
> ./configure
[...]
checking ck_pr.h usability... yes
checking ck_pr.h presence... yes
checking for ck_pr.h... yes
checking for ck_ht_init in -lck... yes
checking for hwloc_topology_init in -lhwloc... yes
checking for jlog_new in -ljlog... no
configure: error: *** libjlog required github.com/omniti-labs/jlog ***

jlog is not present on circonus-labs, it is available at https://labs.omniti.com/jlog/
but this is not what is expected from the message :-)

Looking forward to testing your work 😄

Thomas

Support parameterized logging

Allow logging API to support key value pairs (and perhaps nested K/V pairs).
Allow streams to transform them.
This should support writing fast binary logs as well as something like JSON.

Optionally retain console history.

Allow a config option in the console listener to retain a server-side copy of the command history.

Lua build options don't work.

The --enable-lua configuration option is allowed to take no, lua52 or LuaJIT as options.

If you pass --disable-lua, the build still tries to build the various Lua components. It is clear that no-one ever does this.

If you pass --enable-lua=lua52, you get an unconditional error.

If you pass --enable-lua=LuaJIT, you need an undocumented dependency on the udns-devel package (at least on Fedora).

What's the right approach here? Enforce that LuaJIT is required? Try to make Lua optional? Support multiple different Lua implementations?

Want option to disable trace output files

The run_glider function unconditionally saves the stdout of the configured glider to tracedir. Depending on the glider being used, this can end up being a rather large amount of output, and frequent crashes can cause the tracedir to grow quite large.

The operator should be able to disable this functionality if it is causing issues or provides insufficient value.