GithubHelp home page GithubHelp logo

morganstanley / binlog Goto Github PK

View Code? Open in Web Editor NEW
596.0 596.0 72.0 1.08 MB

A high performance C++ log library, producing structured binary logs

Home Page: http://opensource.morganstanley.com/binlog/

License: Apache License 2.0

CMake 2.49% C++ 97.51%

binlog's People

Contributors

benedekthaler avatar bingenito avatar erenon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

binlog's Issues

Request: Standardise default bread timestamp

Having the default timestamp be in American month/day format and missing the year is a bit confusing. It's also unclear which timezone the logs refer to. Perhaps the default output should be ISO 8601 or similar standardised format?

Compile Errors on macOS 10.15 (Xcode 11.3)

Thank you for the library! Very interesting and would like to try it out, but running into issues compiling the code and unit tests.

Using macOS 10.15 with Xcode 11.3, trying to compile the code with unit tests results in errors:

[ 20%] Building CXX object CMakeFiles/UnitTest.dir/test/unit/mserialize/roundtrip.cpp.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -DBOOST_ALL_DYN_LINK -DBOOST_ALL_NO_LIB -DBOOST_TEST_NO_MAIN -DBOOST_UNIT_TEST_FRAMEWORK_DYN_LINK -I/tmp/build/binlog/bin -I/tmp/build/binlog/include -isystem /usr/local/include  -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk   -Wall -Wextra -Werror -pedantic -Wconversion -Wsign-conversion -Wold-style-cast -Wshadow -std=c++14 -o CMakeFiles/UnitTest.dir/test/unit/mserialize/roundtrip.cpp.o -c /tmp/build/binlog/test/unit/mserialize/roundtrip.cpp
In file included from /tmp/build/binlog/test/unit/mserialize/roundtrip.cpp:6:
In file included from /tmp/build/binlog/include/mserialize/serialize.hpp:4:
/tmp/build/binlog/include/mserialize/detail/Serializer.hpp:39:5: error: static_assert failed due to requirement
      'always_false<std::__1::__bit_const_reference<std::__1::vector<bool, std::__1::allocator<bool> > > >::value' "T is not serializable"
    static_assert(always_false<T>::value, "T is not serializable");
    ^             ~~~~~~~~~~~~~~~~~~~~~~
/tmp/build/binlog/include/mserialize/serialize.hpp:34:32: note: in instantiation of function template specialization
      'mserialize::detail::BuiltinSerializer<std::__1::__bit_const_reference<std::__1::vector<bool, std::__1::allocator<bool> > >, void>::serialize<OutputStream>'
      requested here
  detail::Serializer<T>::type::serialize(in, ostream);
                               ^
/tmp/build/binlog/include/mserialize/detail/Serializer.hpp:133:19: note: in instantiation of function template specialization
      'mserialize::serialize<std::__1::__bit_const_reference<std::__1::vector<bool, std::__1::allocator<bool> > >, OutputStream>' requested here
      mserialize::serialize(elem, ostream);
                  ^
/tmp/build/binlog/include/mserialize/detail/Serializer.hpp:109:5: note: in instantiation of function template specialization
      'mserialize::detail::BuiltinSerializer<std::__1::vector<bool, std::__1::allocator<bool> >, void>::serialize_elems<OutputStream>' requested here
    serialize_elems(is_sequence_batch_serializable<const Sequence>{}, s, size32, ostream);
    ^
/tmp/build/binlog/include/mserialize/serialize.hpp:34:32: note: in instantiation of function template specialization
      'mserialize::detail::BuiltinSerializer<std::__1::vector<bool, std::__1::allocator<bool> >, void>::serialize<OutputStream>' requested here
  detail::Serializer<T>::type::serialize(in, ostream);
                               ^
/tmp/build/binlog/test/unit/mserialize/roundtrip.cpp:45:15: note: in instantiation of function template specialization 'mserialize::serialize<std::__1::vector<bool,
      std::__1::allocator<bool> >, OutputStream>' requested here
  mserialize::serialize(in, ostream);
              ^
/tmp/build/binlog/test/unit/mserialize/roundtrip.cpp:279:3: note: in instantiation of function template specialization '(anonymous
      namespace)::roundtrip_into<std::__1::vector<bool, std::__1::allocator<bool> >, std::__1::vector<bool, std::__1::allocator<bool> > >' requested here
  roundtrip_into(in, out);
  ^
In file included from /tmp/build/binlog/test/unit/mserialize/roundtrip.cpp:6:
In file included from /tmp/build/binlog/include/mserialize/serialize.hpp:4:
/tmp/build/binlog/include/mserialize/detail/Serializer.hpp:44:5: error: static_assert failed due to requirement
      'always_false<std::__1::__bit_const_reference<std::__1::vector<bool, std::__1::allocator<bool> > > >::value' "T is not serializable"
    static_assert(always_false<T>::value, "T is not serializable");
    ^             ~~~~~~~~~~~~~~~~~~~~~~
/tmp/build/binlog/include/mserialize/serialize.hpp:46:39: note: in instantiation of member function
      'mserialize::detail::BuiltinSerializer<std::__1::__bit_const_reference<std::__1::vector<bool, std::__1::allocator<bool> > >, void>::serialized_size' requested here
  return detail::Serializer<T>::type::serialized_size(in);
                                      ^
/tmp/build/binlog/include/mserialize/detail/Serializer.hpp:156:29: note: in instantiation of function template specialization
      'mserialize::serialized_size<std::__1::__bit_const_reference<std::__1::vector<bool, std::__1::allocator<bool> > > >' requested here
      result += mserialize::serialized_size(elem);
                            ^
/tmp/build/binlog/include/mserialize/detail/Serializer.hpp:121:12: note: in instantiation of member function 'mserialize::detail::BuiltinSerializer<std::__1::vector<bool,
      std::__1::allocator<bool> >, void>::sizeof_elems' requested here
         + sizeof_elems(std::is_arithmetic<sequence_data_t<const Sequence>>{}, s);
           ^
/tmp/build/binlog/include/mserialize/serialize.hpp:46:39: note: in instantiation of member function 'mserialize::detail::BuiltinSerializer<std::__1::vector<bool,
      std::__1::allocator<bool> >, void>::serialized_size' requested here
  return detail::Serializer<T>::type::serialized_size(in);
                                      ^
/tmp/build/binlog/test/unit/mserialize/roundtrip.cpp:48:57: note: in instantiation of function template specialization 'mserialize::serialized_size<std::__1::vector<bool,
      std::__1::allocator<bool> > >' requested here
  BOOST_TEST(std::size_t(stream.tellp()) == mserialize::serialized_size(in));
                                                        ^
/tmp/build/binlog/test/unit/mserialize/roundtrip.cpp:279:3: note: in instantiation of function template specialization '(anonymous
      namespace)::roundtrip_into<std::__1::vector<bool, std::__1::allocator<bool> >, std::__1::vector<bool, std::__1::allocator<bool> > >' requested here
  roundtrip_into(in, out);
  ^
2 errors generated.
make[2]: *** [CMakeFiles/UnitTest.dir/test/unit/mserialize/roundtrip.cpp.o] Error 1
make[1]: *** [CMakeFiles/UnitTest.dir/all] Error 2
make: *** [all] Error 2

Thanks!

[INVALID] Syscall param writev(vector[...]) points to uninitialised byte(s)

I've implemented a consume class

#ifndef UTILS_BINLOG_CONSUME_THREAD_HPP
#define UTILS_BINLOG_CONSUME_THREAD_HPP

#include <atomic>
#include <fstream>
#include <string>
#include <thread>

#include "binlog/binlog.hpp"

namespace utils {

/**
* @brief Setup a thread to run continously polling for binlog logging events and write them to the provided file
*/
class BinlogConsumeThread
{
public:
  explicit BinlogConsumeThread(const std::string & file_path);

  BinlogConsumeThread(const BinlogConsumeThread &) = delete;
  BinlogConsumeThread(BinlogConsumeThread &&) noexcept = delete;
  BinlogConsumeThread & operator=(const BinlogConsumeThread &) = delete;
  BinlogConsumeThread & operator=(BinlogConsumeThread &&) noexcept = delete;

  ~BinlogConsumeThread();

private:
  std::ofstream log_file_;
  std::atomic<bool> shut_down_ = false;
  std::thread thread_;
};

} // namespace utils

#endif // UTILS_BINLOG_CONSUME_THREAD_HPP
 #include "utils/binlog_consume_thread.hpp"

namespace utils {

BinlogConsumeThread::BinlogConsumeThread(const std::string & file_path) :
  log_file_{file_path, std::ofstream::out | std::ofstream::binary},
  thread_{[&]() {
	  while(!shut_down_.load(std::memory_order_acquire))
	  {
		  binlog::consume(log_file_);
		  log_file_.flush();
		  std::this_thread::yield();
	  }
  }}
{}

BinlogConsumeThread::~BinlogConsumeThread()
{
  shut_down_.store(true, std::memory_order_release);
  thread_.join();
}

} // namespace utils

which is started from

main(){
   [[maybe_unused]] auto logging_thread = BinLogConsumeThread{"/tmp/log.blog"};
}

and I'm getting valgrind errors on shutdown (details below). Not sure if this an issue within binlog or with my understanding of acceptable ways to call consume from a secondary thread?

==36777== Syscall param writev(vector[...]) points to uninitialised byte(s)
==36777==    at 0x61D7230: writev (in /usr/lib64/libc-2.17.so)
==36777==    by 0x57286C4: std::__basic_file<char>::xsputn_2(char const*, long, char const*, long) (in /usr/lib64/libstdc++.so.6.0.19)
==36777==    by 0x5761C63: std::basic_filebuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib64/libstdc++.so.6.0.19)
==36777==    by 0x5742641: std::ostream::write(char const*, long) (in /usr/lib64/libstdc++.so.6.0.19)
==36777==    by 0x4271EF: binlog::Session::ConsumeResult binlog::Session::consume<std::basic_ofstream<char, std::char_traits<char> > >(std::basic_ofstream<char, std::char_traits<char> >&) (Session.hpp:336)
==36777==    by 0x426F02: binlog::Session::ConsumeResult binlog::consume<std::basic_ofstream<char, std::char_traits<char> > >(std::basic_ofstream<char, std::char_traits<char> >&) (default_session.hpp:53)
==36777==    by 0x4260D3: utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}::operator()() const (binlog_consume_thread.cpp:10)
==36777==    by 0x426605: void std::__invoke_impl<void, utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}>(std::__invoke_other, utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}&&) (invoke.h:60)
==36777==    by 0x4265BA: std::__invoke_result<utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}>::type std::__invoke<utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}>(std::__invoke_result&&, (utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}&&)...) (invoke.h:95)
==36777==    by 0x426567: void std::thread::_Invoker<std::tuple<utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (thread:244)
==36777==    by 0x42653D: std::thread::_Invoker<std::tuple<utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}> >::operator()() (thread:251)
==36777==    by 0x426521: std::thread::_State_impl<std::thread::_Invoker<std::tuple<utils::BinlogConsumeThread::BinlogConsumeThread(std::string const&)::{lambda()#1}> > >::_M_run() (thread:195)
==36777==  Address 0x766ce8f is 159 bytes inside a block of size 1,048,632 alloc'd
==36777==    at 0x4C2ABF8: operator new[](unsigned long) (vg_replace_malloc.c:433)
==36777==    by 0x40B089: binlog::Session::Channel::Channel(binlog::Session&, unsigned long, binlog::WriterProp) (Session.hpp:199)
==36777==    by 0x4243C3: void __gnu_cxx::new_allocator<binlog::Session::Channel>::construct<binlog::Session::Channel, binlog::Session&, unsigned long&, binlog::WriterProp>(binlog::Session::Channel*, binlog::Session&, unsigned long&, binlog::WriterProp&&) (new_allocator.h:147)
==36777==    by 0x4226E3: void std::allocator_traits<std::allocator<binlog::Session::Channel> >::construct<binlog::Session::Channel, binlog::Session&, unsigned long&, binlog::WriterProp>(std::allocator<binlog::Session::Channel>&, binlog::Session::Channel*, binlog::Session&, unsigned long&, binlog::WriterProp&&) (alloc_traits.h:484)
==36777==    by 0x41FF27: std::_Sp_counted_ptr_inplace<binlog::Session::Channel, std::allocator<binlog::Session::Channel>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<binlog::Session&, unsigned long&, binlog::WriterProp>(std::allocator<binlog::Session::Channel>, binlog::Session&, unsigned long&, binlog::WriterProp&&) (shared_ptr_base.h:548)
==36777==    by 0x41BC38: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<binlog::Session::Channel, std::allocator<binlog::Session::Channel>, binlog::Session&, unsigned long&, binlog::WriterProp>(binlog::Session::Channel*&, std::_Sp_alloc_shared_tag<std::allocator<binlog::Session::Channel> >, binlog::Session&, unsigned long&, binlog::WriterProp&&) (shared_ptr_base.h:679)
==36777==    by 0x417DC9: std::__shared_ptr<binlog::Session::Channel, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<binlog::Session::Channel>, binlog::Session&, unsigned long&, binlog::WriterProp>(std::_Sp_alloc_shared_tag<std::allocator<binlog::Session::Channel> >, binlog::Session&, unsigned long&, binlog::WriterProp&&) (shared_ptr_base.h:1344)
==36777==    by 0x414FFA: std::shared_ptr<binlog::Session::Channel>::shared_ptr<std::allocator<binlog::Session::Channel>, binlog::Session&, unsigned long&, binlog::WriterProp>(std::_Sp_alloc_shared_tag<std::allocator<binlog::Session::Channel> >, binlog::Session&, unsigned long&, binlog::WriterProp&&) (shared_ptr.h:359)
==36777==    by 0x411FDE: std::shared_ptr<binlog::Session::Channel> std::allocate_shared<binlog::Session::Channel, std::allocator<binlog::Session::Channel>, binlog::Session&, unsigned long&, binlog::WriterProp>(std::allocator<binlog::Session::Channel> const&, binlog::Session&, unsigned long&, binlog::WriterProp&&) (shared_ptr.h:702)
==36777==    by 0x40E930: std::shared_ptr<binlog::Session::Channel> std::make_shared<binlog::Session::Channel, binlog::Session&, unsigned long&, binlog::WriterProp>(binlog::Session&, unsigned long&, binlog::WriterProp&&) (shared_ptr.h:718)
==36777==    by 0x40B2AC: binlog::Session::createChannel(unsigned long, binlog::WriterProp) (Session.hpp:241)
==36777==    by 0x40B885: binlog::SessionWriter::SessionWriter(binlog::Session&, unsigned long, unsigned long, std::string) (SessionWriter.hpp:126)

[INVALID] a crash from binlog

image

Hi,
I got this crash when I try to use writer & session with TSC.
My implementation of binlog as below

class BinLoggerBinary
{
public:
	BinLoggerBinary(std::ostream& binary) :_binary(binary),
		_severity(binlog::Severity::trace),
		_writer(_session),
		_filter([this](const binlog::EventSource& source) {
		return source.severity >= this->_severity;
			})
	{
		// Create a ClockSync, that connects the TSC value to the wall clock time.
		const auto tscValue = __rdtsc();
		const binlog::ClockSync systemClockSync = binlog::systemClockSync();
		const binlog::ClockSync tscSync{
			tscValue,
			tscFrequency(),
			systemClockSync.nsSinceEpoch,
			systemClockSync.tzOffset,
			systemClockSync.tzName
		};

		// Add the created ClockSync to the metadata of the session.
		// Events consumed after this are assumed to be timestamped
		// with the clock defined by the ClockSync (this case, TSC)
		_session.setClockSync(tscSync);
	}
	std::uint64_t tscFrequency()
	{
#if defined(__linux__)
		long freq;

		// If the kernel is exporting the tsc frequency use that. There are issues
		// where cpuinfo_max_freq cannot be relied on because the BIOS may be
		// exporintg an invalid p-state (on x86) or p-states may be used to put the
		// processor in a new mode (turbo mode). Essentially, those frequencies
		// cannot always be relied upon. The same reasons apply to /proc/cpuinfo as
		// well.
		if (ReadFromFile("/sys/devices/system/cpu/cpu0/tsc_freq_khz", &freq)
			// If CPU scaling is in effect, we want to use the *maximum* frequency,
			// not whatever CPU speed some random processor happens to be using now.
			|| ReadFromFile("/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq",
				&freq)) {
			// The value is in kHz (as the file name suggests).  For example, on a
			// 2GHz warpstation, the file contains the value "2000000".
			return freq * 1000.0;
		}
		else
		{
			throw std::runtime_error("Not found tsc freq in this OS,make sure cpuinfo_max_freq exist.");
			return 0;
		}
#else
		// In NT, read MHz from the registry. If we fail to do so or we're in win9x
		// then make a crude estimate.
		DWORD data, data_size = sizeof(data);
		if (IsWindowsXPOrGreater() &&
			SUCCEEDED(
				SHGetValueA(HKEY_LOCAL_MACHINE, "HARDWARE\\DESCRIPTION\\System\\CentralProcessor\\0", "~MHz", nullptr, &data, &data_size)))
			return ((uint64_t)data *
				(uint64_t)(1000 * 1000));  // was mhz
		else
		{
			throw std::runtime_error("Not found tsc freq in this OS,make sure HARDWARE\\DESCRIPTION\\System\\CentralProcessor\\0 exist.");
			return 0;
		}
#endif
	}
	void consume()
	{
		_session.consume(_binary);
		_binary.flush();
	}

	binlog::SessionWriter& GetWriter()
	{
		return _writer;
	}

	void set_log_level(binlog::Severity level)
	{
		_severity = level;
	}
private:		
	std::ostream& _binary;
	binlog::Session _session;
	binlog::Severity _severity;
	binlog::SessionWriter _writer;		
	binlog::EventFilter _filter;		
};

Allow multiple destinations for Session::consume

Session::consume takes an arbitrary object modeling the mserialize::OutputStream concept, and writes metadata and data to it.

This decouples log rotation and destination selection from the Session. To send the data to multiple destinations, one simply needs to create a MultiOutputStream, which forwards calls to write to wrapped streams, e.g:

class MultiOutputStream {
public:
  MultiOutputStream(/* ... */);

  MultiOutputStream& write(const char* buf, std::streamsize size)
  {
    _file.write(buf, size);
    _monitor.write(buf, size);
    _console.write(buf, size);
    return *this;
  }

private:
  std::ostream _file;
  Monitoring _monitor;
  Console _console;
};

Writing a generic version of the above (e.g: MultiOutputStream<Streams...>) is non-trivial (e.g: how does the constructor look like, how to expose stats of children, etc), requires a lot of complexity, and probably cannot cover some use-cases. A specialized solution for ones use-case could be simpler and easier to use.

Therefore, unless a better design is suggested, create an example that shows how to create and use such a multi stream.

Make bread able to sort the logstream by time

By default, bread prints log events in the log stream in order of appearance. Normally, this means the events of each thread are batched together, making cross-thread interaction more difficult to reason about.

Add a switch (-s) to bread, that makes it sort events in the stream by time. A simple implementation would be to read every event, then sort and print them in one go. However, this approach has a high time-to-first-event latency, high memory usage, and does not work with live streams.

A more involved approach would be to assume that events are sorted to a certain extent (e.g: events more than 60 seconds apart in the log will not be out of order), read a window of events sort and print some of the window, then repeat.

Document limitations of logging in static ctor/dtor contexts

When binlog::default_session is used in static constructors or destructors, logging can trigger UB, depending on relative initialization order, as shown in: #90 .
Possible fixes include:

  • force initialization of the binlog Session before other static logging objects (e.g: by passing a ref of the Session to the latters constructor)
  • use nifty counter (complicated)

Document this limitation in UserGuide.md and in default_session.hpp.

Non standalone builds should mark include files as SYSTEM

The binlog CMake target_include_directories call should detect when binlog is being compiled as part of another project and mark the include files as SYSTEM so that extra compiler warnings enabled by the parent project can't be triggered on the binlog header files.

Reasoning: The current convention in binlog of having constructor parameters named the same as private class members is causing compiler warnings for users who have -Wshadow enabled and breaking users who have both -Wshadow and -Werror enabled.

Example Fix: https://github.com/microsoft/GSL/blob/master/CMakeLists.txt#L58

Narrowing conversion error on Windows.

Hello,

I'm trying out the library on Windows and it seems that I couldn't get it to work properly. Every time I try to compile the helloworld example, I got the error C2397: conversion from 'long' to 'uint64_t' requires a narrowing conversion at the BINLOG_INFO("Hello {}!", "World"); line.

However, when I compile the same code with WSL (GCC 8.3.0) everything works as expected, no warning at all. I also was able to build the helloworld example in the CMake generated project, and it ran just fine with the same VS.

I'm using Visual Studio 2019 on Windows 10.

I might be missing something obvious but I couldn't figure it out. Thank you so much for your time and help.

C-strings support

I've noticed that char* is understood as a pointer to char, but not as common c-string char const*.
For example, strerror(int) function returns a char* string.
I would consider this as a bug, but not a feature request.

Don't force build type on non standalone builds

For standalone builds of binlog it's probably fine to force an unspecified CMAKE_BUILD_TYPE to be RelWithDebInfo. However setting this value in a non standalone build affects all other projects which are being built.

Document and test how to tail live logfiles

It is a common use-case to read the logfile of a still running application. It is convenient to see the new events as they are written automatically, e.g: by using tail -f logfile. Similar effect can be achieved with binary logfiles:

tail -c0 -f logfile.blog | bread

Document this trick and make sure via testing that it works (and will continue to work), even if the pipe contains partial events temporarily.

Add pretty printing support to bread

The default text representation of some types that bread uses (i.e: Typename{ field: value }) is not human readable for some types. For example, logging std::chrono::time_point as a uint64_t is efficient, but not human readable. Showing the type of std::filesystem::path adds too much noise. Memory addresses should be shown as hex, instead of decimal, etc.

Add a pretty printer mechanism to bread, that changes the text representation of well known types.

Allow log rotation

Log rotation is a popular feature, allowing long running apps to keep logging while discarding old logs.

Session::consume(ostream&) takes an arbitrary ostream, therefore allows the target to be rotated by simply changing the given argument. However, since the session does not know about the target file, it does not re-send the metadata: the rotated files can be converted only together, which does not allow the removal of old files.

Add a new API (e.g: Session::rewindMetadata) which makes the next consume to emit metadata again, allowing simple rotations by a change of the consume argument.

Add support for std::error_code

Make std::error_code loggable:

std::error_code ec;
BINLOG_INFO("{}", ec);

Trivial solution is to make it loggable in terms of ec.message().

(Alternative solution: check if ec.category() is system category,
then log ec.value() (and pretty print it in the reader),
otherwise log ec.message().)

[INVALID] Convert to text file using bread on Windows.

Hi,

Thank you so much for the great library. Everything is working well so far.

However, I'm having problem converting the binary log file into text file using bread. The binary log is just from the hello world sample in the project. I can read the log just fine using the command line. But whenever I write to write the log back to a text file using: .\bread.exe .\hello.blog > .\hello.txt, the content of hello.txt is just binary data. I'm using Windows 10 and built the library with VS19 with the current master branch.

I also tried and used WSL to build the library under Ubuntu with GCC 9.3 and used bread in there too. The result is still the same, the text file only contain the binary data not readable text.

Is there any other step that I need to do? I'm so sorry if I missed something but I couldn't find any more information in the document site. The attached zip file is the binary log and the text converted using bread. I don't know whether this can help with anything as it's just the output from the sample program.

Output.zip

Thank you so much for your time and help.

Recursive BINLOG_ADAPT_STRUCT for derived classes

Hi,

Is there a recursive way to setup logging of derived classes that doesn't require having to repeatedly call the base class members/getters? Repeating base class members/getters in multiple derived classes can get quite cumbersome and results in lots of changes and easy to make errors if the base class is modified, e.g.,

BINLOG_ADAPT_STRUCT(Base, a, b)
BINLOG_ADAPT_STRUCT(Derived, c, d, Base)

instead of

BINLOG_ADAPT_STRUCT(Base, a, b)
BINLOAD_ADAPT_STRUCT(Dervied, c, d, a, b)

for

struct Base
{
   int a, b;
};
struct Derived : public Base
{
   int c,d;
};

Allow writing text directly

A key feature of binlog is that it writes binary logfiles, that are smaller, cheap to create and easy to mine. However, some applications might want to trade performance and flexibility to convenience, and get text logs directly from the application, e.g on stdout, equivalent to:

$ ./my_app | bread

but without using bread. This is possible by creating a special stream, e.g: TextOutputStream, that converts received binlog data to text, on the fly. Such a stream would need to link to bread logic, therefore it requires a linkable binlog library to be exposed by cmake.

Expected usage:

binlog::TextOutputStream tos(std::cout); // also takes optional format and date format
session.consume(tos); // text log appear on stdout

support fixed size array, and struct copy

To further improve the performance, it's better to support fixed size array - std::array<T, N>, char [N]. You can embed the size of the array in the tag, so in most cases, a simple memcpy will be enough for serialization just like primitive types do.

suggested tag:

<FixedSizeArrayTag> := # `SizeInHex' <ElementTypeTag>

For simple POD struct (which may contain char[N] as fields), we'd also like to have a memcpy style tag. e.g.

<MemcpyStructTag> :=
={SizeInHex [`OffsetInHex' <FieldTypeTag>]*}

If the above two can be implemented, the serialization will be effectively optimized by the compiler to a fixed-size-memcpy in a lot of cases, thus push the performance to the limit.

Default SessionWriter id to unique value

Users which use the default macros BINLOG_INFO etc. don't get any data printed for "Writer (thread) id" from bread and therefore are unable to distinguish log lines from different threads. This is because the SessionWriter constructor defaults id to 0. It would probably make sense for the default value to be std::this_thread::get_id() or similar?

Native support for logging pointer addresses

Currently attempting to pass a void * pointer as an argument results in a compile error requiring the user to do a reinterpret_cast to std::uintptr_t or similar. It would be great if this cast could be moved into the logging framework so that the user isn't required to do it. Even better, binlog could treat pointers as first class citizens by having a new mserialize tag so that bread would log them by default in 0x format, similar to printf.

Add support for std::optional

The below seems to be missing from mserialize and needs to be added manually by the user

namespace mserialize { namespace detail {
  template <typename T> struct is_optional<std::optional<T>> : std::true_type {};
}}

T[N] like fields are not supported for struct tag

The Problem:

Suppose you have a struct like:

struct ABC {
  int a;
  short b[10];
  char c[12][2];
};

When you do

MSERIALIZE_MAKE_STRUCT_TAG(ABC, a, b, c);

There will be an error complaining you cann't use a T[] as return type in serializable_member_type.

The Fix:

Add the following overloads to make_struct_tag.hpp

template<typename T, typename E, int N>
auto serializable_member_type(E(T::*)[N]) -> std::vector<E>;

template<typename T, typename E, int N, int M>
auto serializable_member_type(E(T::*)[N][M]) -> std::vector<std::vector<E>>;

template<typename T, typename E, int N, int M, int K>
auto serializable_member_type(E(T::*)[N][M][K]) -> std::vector<std::vector<std::vector<E>>>;

In this way, the fields will be correctly mathed as a (possibly nested) Sequence.

I have tested with tag/serialization/deserialization/visit, they all seems to be ok.

Fix MSVC warnings

There are only a few, harmless warnings, but in header files, affecting the clients build experience.

Add TSC example

By default, timestamps are created by std::chrono::system_clock. There's a faster time source, TSC, with its own issues. Create an example that shows how the interested low latency user can use TSC to timestamp events, by adding a custom ClockSync.

bread ubsan error

binlog/include/binlog/Time.cpp:7:41: runtime error: signed integer overflow: 31534085395 * 1000000000 cannot be represented in type 'long int'

Invalid iterator comparison on shutdown

With g++ debug checking enabled (-D_GLIBCXX_SANITIZE_VECTOR -D_GLIBCXX_DEBUG -D_GLIBCXX_DEBUG_PEDANTIC") the following error is encountered on shutdown.

/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/debug/safe_iterator.h:552:
    error:
    attempt
    to
    compare
    a
    singular

    iterator
    to
    a
    past-the-end

    iterator
    .

Objects involved in the operation:
iterator "lhs" @ 0x0x287e2e8 {
type = ▒hI1▒ (mutable iterator);
  state = singular;
  references sequence with type `NSt7__debug6vectorISt10shared_ptrIN6binlog7Session7ChannelEESaIS5_EEE' @ 0x0x287e2e8
}
iterator "rhs" @ 0x0x287e2e8 {
type = ▒hI1▒ (mutable iterator);
  state = past-the-end;
  references sequence with type `NSt7__debug6vectorISt10shared_ptrIN6binlog7Session7ChannelEESaIS5_EEE' @ 0x0x287e2e8
}

Empty container triggers UBSan

BINLOG_INFO("{}", std::vector<int>{});
binlog/detail/QueueWriter.hpp:78:38: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:43:28: note: nonnull attribute specified here```

Feature Request: Support collections with iterators of type std::input_iterator_tag

As above, would it be possible to support collections with iterators of std::input_iterator_tag instead of requiring std::forward_iterator_tag?

It would mean it's not possible to know the size of the collection beforehand and an alternative way of encoding (removing the explicit serializing of the number of elements?) would be required. This might be useful in the generic case as well so as to avoid iterating through collections twice.

Enable -Wshadow compiler warning

Regardless the benefits, make sure no symbol is shadowed, even at the cost of readability changes (e.g: where constructor arguments shadow members of the same name)

Feature request: structural logging interface

Structural logging is more friendly to programs. With tags encodes the structual information and binlogs contains the value, it's possible and easy to convert binlogs to json (or other data formats), which is more suitable for further processing by other people.

It's best to have an interface like:

LOG({'key1', Value1}, {'key2', Value2}, ...);

Is it possible?

Heap use after free on shutdown

Integrated BINLOG_INFO logging into google benchmark based tests and receive the following on shutdown.

WRITE of size 1 at 0x60600000d830 thread T0
    #0 0x68fcc5 in std::__atomic_base<bool>::store(bool, std::memory_order) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/atomic_base.h:374:2
    #1 0x68fcc5 in std::__atomic_base<bool>::operator=(bool) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/atomic_base.h:267
    #2 0x68fa0d in std::atomic<bool>::operator=(bool) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/atomic:79:22
    #3 0x680c1d in binlog::SessionWriter::~SessionWriter() /Morgan-Stanley_binlog/20200306_0c86ccd/include/binlog/SessionWriter.hpp:135:22
    #4 0x7efc08019e08  (/lib64/libstdc++.so.6+0x5ce08)
    #5 0x7efc072faa68 in __run_exit_handlers (/lib64/libc.so.6+0x38a68)
    #6 0x7efc072faab4 in __GI_exit (/lib64/libc.so.6+0x38ab4)
    #7 0x7efc072e3c0b in __libc_start_main (/lib64/libc.so.6+0x21c0b)
    #8 0x512ea8 in _start (perf+0x512ea8)

0x60600000d830 is located 16 bytes inside of 56-byte region [0x60600000d820,0x60600000d858)
freed by thread T0 here:
    #0 0x6452e0 in operator delete(void*) (perf+0x6452e0)
    #1 0x687667 in __gnu_cxx::new_allocator<std::_List_node<binlog::Session::Channel> >::deallocate(std::_List_node<binlog::Session::Channel>*, unsigned long) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/ext/new_allocator.h:125:2
    #2 0x687614 in std::allocator_traits<std::allocator<std::_List_node<binlog::Session::Channel> > >::deallocate(std::allocator<std::_List_node<binlog::Session::Channel> >&, std::_List_node<binlog::Session::Channel>*, unsigned long) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/alloc_traits.h:462:13
    #3 0x68621e in std::_List_base<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::_M_put_node(std::_List_node<binlog::Session::Channel>*) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:454:9
    #4 0x685f23 in std::_List_base<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::_M_clear() /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/list.tcc:81:4
    #5 0x685be1 in std::_List_base<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::~_List_base() /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:507:9
    #6 0x681624 in std::list<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::~list() /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:835:23
    #7 0x680e18 in binlog::Session::~Session() /Morgan-Stanley_binlog/20200306_0c86ccd/include/binlog/Session.hpp:46:7
    #8 0x7efc072faa68 in __run_exit_handlers (/lib64/libc.so.6+0x38a68)

previously allocated by thread T0 here:
    #0 0x644350 in operator new(unsigned long) (perf+0x644350)
    #1 0x68a4cc in __gnu_cxx::new_allocator<std::_List_node<binlog::Session::Channel> >::allocate(unsigned long, void const*) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/ext/new_allocator.h:111:27
    #2 0x68a430 in std::allocator_traits<std::allocator<std::_List_node<binlog::Session::Channel> > >::allocate(std::allocator<std::_List_node<binlog::Session::Channel> >&, unsigned long) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/alloc_traits.h:436:20
    #3 0x689ee6 in std::_List_base<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::_M_get_node() /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:450:16
    #4 0x6899b3 in std::_List_node<binlog::Session::Channel>* std::list<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::_M_create_node<binlog::Session&, unsigned long&, binlog::WriterProp>(binlog::Session&, unsigned long&, binlog::WriterProp&&) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:642:21
    #5 0x689393 in void std::list<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::_M_insert<binlog::Session&, unsigned long&, binlog::WriterProp>(std::_List_iterator<binlog::Session::Channel>, binlog::Session&, unsigned long&, binlog::WriterProp&&) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:1903:18
    #6 0x688bb6 in binlog::Session::Channel& std::list<binlog::Session::Channel, std::allocator<binlog::Session::Channel> >::emplace_back<binlog::Session&, unsigned long&, binlog::WriterProp>(binlog::Session&, unsigned long&, binlog::WriterProp&&) /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/stl_list.h:1235:10
    #7 0x687f84 in binlog::Session::createChannel(unsigned long, binlog::WriterProp) /Morgan-Stanley_binlog/20200306_0c86ccd/include/binlog/Session.hpp:241:13
    #8 0x699e3c in binlog::SessionWriter::replaceChannel(unsigned long) /Morgan-Stanley_binlog/20200306_0c86ccd/include/binlog/SessionWriter.hpp:203:46
    #9 0x738732 in bool binlog::SessionWriter::addEvent<std::string const&, unsigned long, unsigned long, std::basic_string_view<char, std::char_traits<char> >, unsigned long, unsigned int>(unsigned long, unsigned long, std::string const&, unsigned long&&, unsigned long&&, std::basic_string_view<char, std::char_traits<char> >&&, unsigned long&&, unsigned int&&) /Morgan-Stanley_binlog/20200306_0c86ccd/include/binlog/SessionWriter.hpp:179:5
    #10 0x73597d in void binlog::detail::addEventIgnoreFirst<binlog::SessionWriter, char const (&) [104], std::string const&, unsigned long, unsigned long, std::basic_string_view<char, std::char_traits<char> >, unsigned long, unsigned int>(binlog::SessionWriter&, unsigned long, unsigned long, char const (&) [104], std::string const&, unsigned long&&, unsigned long&&, std::basic_string_view<char, std::char_traits<char> >&&, unsigned long&&, unsigned int&&) /Morgan-Stanley_binlog/20200306_0c86ccd/include/binlog/create_source_and_event.hpp:95:10
    #11
	
	snip
	
    #24 0x7efc072e3c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

SUMMARY: AddressSanitizer: heap-use-after-free /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/atomic_base.h:374:2 in std::__atomic_base<bool>::store(bool, std::memory_order)
Shadow bytes around the buggy address:
  0x0c0c7fff9ab0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c7fff9ac0: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff9ad0: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c7fff9ae0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c7fff9af0: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
=>0x0c0c7fff9b00: fa fa fa fa fd fd[fd]fd fd fd fd fa fa fa fa fa
  0x0c0c7fff9b10: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c7fff9b20: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c7fff9b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff9b40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff9b50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==30054==ABORTING

How to use rdtsc to get time

It's described in the README.md that rdtsc is the fastest way to get time under linux.

Benchmark Std. Dev. Median
One integer (no clock) 0 ns 8 ns
One integer (TSC clock) 0 ns 9 ns
One integer (sys clock) 0 ns 34 ns
One integer (poison cache) 8 ns 740 ns
One string 0 ns 38 ns
Three floats 0 ns 30 ns

But in the soruce code I saw clockNow uses clock_gettime. Is there a way or special version that uses rdtsc to get time under linux?

Thanks.

Multiple loggers, file/line,tid metadata (was: Great tool)

Hi, thanks for open sourcing this tool, bringing the logging framework used in low-latency trading system to the public.

Had a quick look at existing features, any plan to support below two features?

  1. multiple loggers in the same process: we want to separate logs of peripheral threads from critical thread. spdlog enables us to create multiple loggers. How is it supported in binlog?
  2. usage sugar, for example, spdlog's SPDLOG_INFO will log the file name, line number, tid etc.

Thanks!

Add crash recovery

Binlog does asynchronous logging, therefore if the application crashes, the most recent, probably most relevant logs, will still be in the queues, not flushed to the log output. However, if there's a coredump (minidump), nothing is lost, only the queue content must be recovered.

Create a program, brecovery, which takes a coredump (minidump, or any kind of memory image), and extracts unconsumed queue data. Usage:

$ brecovery myapp.1234.core recovered_log.blog

Solution: Make Queue instances discoverable in the cordump, and link queue buffers to queue instances. Steps:

  • Add a const magic number to the Queue class (e.g: a random generated number)
  • Add a instance unique identifier to each queue object (i.e: the pointer of itself)
  • Verify by static_assert the offset of the important Queue fields
  • Allocate slightly larger buffer for each queue. In the beginning of the queue, add a different magic number, and a pointer to the queue.

Recovery process: scan the whole coredump, looking for the two magic numbers. Link found queues and buffers by the queue identifier. Read reader and writer positions from the queue, extract data from the linked buffer accordingly, write extracted data to the specified output.

Testing: on Linux, if available, gcore can be used to create corefiles out of test programs. On Windows, Userdump.exe, ProcDump, or MiniDumpWriteDump can be used.

fails to build in Debug as well as Release mode on g++-10 Linux

It fails to build with the given steps in Debug as well as Release modes on Linux g++-10 with following error:

Scanning dependencies of target TextOutput
make[2]: Leaving directory 'RED/binlog/Release'
make  -f CMakeFiles/TextOutput.dir/build.make CMakeFiles/TextOutput.dir/build
make[2]: Entering directory 'RED/binlog/Release'
[ 29%] Building CXX object CMakeFiles/TextOutput.dir/example/TextOutput.cpp.o
/usr/bin/c++   -IRED/binlog/include  -O2 -g -DNDEBUG   -Wall -Wextra -Werror -pedantic -Wconversion -Wsign-conversion -Wold-style-cast -Wsuggest-override -Wshadow -std=c++14 -o CMakeFiles/TextOutput.dir/example/TextOutput.cpp.o -c RED/binlog/example/TextOutput.cpp
In file included from RED/binlog/include/binlog/Severity.hpp:4,
                 from RED/binlog/include/binlog/Entries.hpp:5,
                 from RED/binlog/include/binlog/EventStream.hpp:4,
                 from RED/binlog/include/binlog/TextOutputStream.hpp:4,
                 from RED/binlog/example/TextOutput.cpp:1:
RED/binlog/include/mserialize/cx_string.hpp: In instantiation of ‘constexpr mserialize::cx_string<N>::cx_string(const char*) [with long unsigned int N = 0]’:
RED/binlog/include/mserialize/cx_string.hpp:95:10:   required from ‘constexpr auto binlog::detail::concatenated_tags(Unused&&, T&& ...) [with Unused = const char (&)[19]; T = {}]’
RED/binlog/example/TextOutput.cpp:8:3:   required from here
RED/binlog/include/binlog/create_source_and_event.hpp:67:31:   in ‘constexpr’ expansion of ‘mserialize::cx_strcat<>()’
RED/binlog/include/mserialize/cx_string.hpp:36:31: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
   36 |     for (std::size_t i = 0; i < N; ++i)
      |                             ~~^~~
cc1plus: all warnings being treated as errors
CMakeFiles/TextOutput.dir/build.make:82: recipe for target 'CMakeFiles/TextOutput.dir/example/TextOutput.cpp.o' failed
make[2]: *** [CMakeFiles/TextOutput.dir/example/TextOutput.cpp.o] Error 1
make[2]: Leaving directory 'RED/binlog/Release'
CMakeFiles/Makefile2:203: recipe for target 'CMakeFiles/TextOutput.dir/all' failed
make[1]: *** [CMakeFiles/TextOutput.dir/all] Error 2
make[1]: Leaving directory 'RED/binlog/Release'
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.