mattiasflodin / reckless Goto Github PK

writer::write can return ERROR_TRY_LATER or ERROR_GIVE_UP but these are not honored by the background thread.

crash_handler_win32.cpp/crash_handler_unix.cpp are missing in CMakeListst.txt? Is it intentional? Why?

The library consistently crashes when example from README is built with it. All you need to change is print the s string in the loop (in addition to integer i) and increase the counter:

    for(int i=0; i!=10000000; ++i) { // <- Increase the counter to crash (change 1 of 2)
        reckless::scoped_indent indent;
        g_log.warn("Warning: %s, %d", s, i);    // <- Print s string to crash (change 2 of 2)
    }

• Verified on Ubuntu 18.04 / gcc 9.5.0, also with gcc 8.3.0
• The crash usually occurs at random counter number.
• The crash seems to occur sooner when stdout_writer is used, later with file_writer.
• Crashing stops as soon as string s gets longer than 16 characters.

Hello,

Is there option to set core affinity for worker thread?

we use cmake to build the lib

cmake -DCMAKE_C_COMPILER=clang ../ 

and run make

it throws out

clang: error: linker command failed with exit code 1 (use -v to see invocation)  

Hi Mattias,

I build a benchmark to compare some Logger in Java, these are Log4J2, logback, spdlog, glog, g3log and reckless.

What I have seen, that spdlog, reckless and g3log are nearly insane, but g3log and reckless have some issues.

The issue I have is a corrupted double-linked list, my problem is, that I have no clou why and when this appears, but it appears sometimes.

*** Error in `java': corrupted double-linked list: 0x00007f56c4046530 ***

Therefore the results I gain from reckless are unusable, like g3log.
But in terms of speed it is similar to spdlog.

As I described, that I use Java I need to mention here, that I use JNI to load it into Java.
This is how I create the logger for reckless, maybe it can be done better:

using log_t = reckless::severity_log<
reckless::indent<4>, // 4 spaces of indent
' ', // Field separator
reckless::severity_field, // Show severity marker (D/I/W/E) first
reckless::timestamp_field // Then timestamp field
>;

reckless::file_writer writer("logs/reckless_async.log");
log_t logger(&writer);

Cheers
Mader102

I was able to build the library with the make file, however when you try to compile the examples or include reckless in another project the following error is thrown:

g++ -std=c++11 severity_log.cpp -I/home/n/git/cpp/reckless/reckless/include -L/home/n/git/cpp/reckless/reckless/lib -lreckless
In file included from /home/n/git/cpp/reckless/reckless/include/reckless/policy_log.hpp:25:0,
                 from /home/n/git/cpp/reckless/reckless/include/reckless/severity_log.hpp:25,
                 from severity_log.cpp:22:
/home/n/git/cpp/reckless/reckless/include/reckless/basic_log.hpp:31:43: fatal error: boost_1_56_0/lockfree/queue.hpp: No such file or directory
compilation terminated.

Hello,
Do you know about Conan?
Conan is modern dependency manager for C++. And will be great if your library will be available via package manager for other developers.

Here you can find example, how you can create package for the library.

If you have any questions, just ask :-)

If you create a log object and open it with a thread_input_buffer_size value of X, perform some logging, close the log, and reopen it with a new value Y for thread_input_buffer size, then the existing thread-local buffer sizes are not adjusted to match the new size. The size is only taken into account when the thread-local buffer is accessed for the first time.

Creating a new log object is fine, as this allocates a new TLS variable.

Hi,

I'm looking to make a single logger that can log to both stderr and to a file at the same time. Based on my brief inspection of basic_log, I realize that I could achieve this by modifying basic_log to take in a vector of writer* pointers rather than a single pointer.

Is this the "right" way to handle logging to multiple files with this library? Is there a better way?

Hello, Mattias!

Thanks for your library, it is a great piece of work one could learn a lot from by reading the source code. I am studying the implementation of MPSC ring buffer now, and I am not sure I got everything right about mpsc_ring_buffer::init(std::size_t). Would you mind clarifying the confusion and also correct me if I got something wrong about the general idea of the following code?

capacity = round_capacity(capacity);

#if defined(__linux__)
    // allocate shared memory for buffer data
    int shm = shmget(IPC_PRIVATE, capacity, IPC_CREAT | S_IRUSR | S_IWUSR);
    if(shm == -1)
        throw std::bad_alloc();

    void* pbase = nullptr;
    while(!pbase) {
        // ensure we can acquire a set of consecutive pages twice as big as required capacity located
        // in process address space;
        pbase = mmap(nullptr, 2*capacity, PROT_NONE,
                MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
        if(MAP_FAILED == pbase)
            throw std::bad_alloc();
        // unmap it - we are not going to use it anyway
        munmap(pbase, 2*capacity);
        // attach shared segment to the location we got from mmap()
        pbase = shmat(shm, pbase, 0);
        // THAT IS WHERE IT FEELS OFF
        // shmat() returns either a pointer to attachment address or ((void *) -1), 
        // but does not return NULL in any case
        // Let's say someone had a chance to occupy the memory location we got from mmap() before we get here
        // (freshly loaded shared library etc.) so shmat() returned ((void *) -1)
        // SHOULDN'T IT BE if (pbase != (void*) -1) ?
        if(pbase) {
            // if first shmat() call succeded , map the same shared segment to the second half of page set
            if((void*)-1 == shmat(shm, static_cast<char*>(pbase) + capacity, 0))
            {
                shmdt(pbase);
                pbase = nullptr;
            }
        }
    }
    // mark segment for removal
    shmctl(shm, IPC_RMID, nullptr);
#elif defined(_WIN32)
// ...
#endif

Thank you again.

Reckless wrongly considers the notify_on_recovery/fail_immediately error policies for temporary/permanent errors to be indicative of whether an error is "fatal" (i.e. permanent) or not. If the error policy for temporary errors is fail_immediately then writer errors will be thrown as fatal_flush_error and the background thread is shut down. This is bad if you're interested in immediate delivery of errors in the form of exceptions, but still wish to continue using the logger.

We should offer a variant of write() that writes to an error_code and does not throw exceptions, and we shouldn't terminate the background thread just because the policy is fail_immediately. We can terminate the thread on permanent errors, but the important thing is that fail_immediately should only control whether basic_log::fatal_error_code (which should be renamed) is set or not.

See e.g. https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions and __ATOMIC_HLE_ACQUIRE flag in GCC.

For AMD: https://en.wikipedia.org/wiki/Advanced_Synchronization_Facility

How does this library respond to scalability? One example is multi-process logging (Possibly, multiple processes writing on the same file).

Thanks!

This is obviously something that will reoccur if updating isn't automated, so some sort of release script is needed.

Hi,

SIGUNUSED is synonymous with SIGSYS, and is no longer available since glibc version 2.26 and is causing build errors. I think it can be removed from the signals initializer_list<> in reckless/src/crash_handler.cpp.

Thanks.

it looks like that you are missing a -pthread here: https://github.com/mattiasflodin/reckless#building
Also -lreckless looks wrong it must be -lasynclog, because your Makefile creates libasynclog.a.

Code:

    std::string str("~~~~~~~~~~~~~~~");

    for (int i = 0; i != 1000000; ++i)
      g_log.info("Info line: %s @%d", str, i);

Result in:

munmap_chunk(): invalid pointer
Aborted

Randomly I saw also:

munmap_chunk(): invalid size

If str has more than 15 characters problem doesn't appear. @mattiasflodin Could you take a look at that? Probably it is similar to #35 (also short string).

Would you like to replace any double quotes by angle brackets around file names for include statements?

Now I see some boost libs are located in boost/boost_1_56_0 and boost is included like #include <boost_1_56_0/lockfree/queue.hpp> https://github.com/mattiasflodin/reckless/blob/master/reckless/include/reckless/basic_log.hpp#L31 .
But common practice is to include boost like #include <boost/lockfree/queue.hpp>.

So, I suggest to move all boost stuff to boost/boost_1_56_0/boost, add include paths to boost/boost_1_56_0 and all includes will be just like #include <boost/lockfree/queue.hpp>.

This purpose of this is to use other (system, newer) versions of boost rather than 1.56.0.

Hi! Thank you for sharing this awesome library!

I need to create a JSON logger that could accept custom JSON fields for context logging. So apart from the message, severity etc there would be other, custom fields.

Is this possible with reckless?

Thanks!

use vs2019 /mtd But report an error

Due to Windows memory management limitations we set the input buffer to 64 kb. But it should be possible to have a separate capacity that is smaller than the size of the ring buffer, in order to reduce variance in latency during high load.

When basic_log copies client-provided arguments to the log queue, it does not guard against exceptions.

Makefile.conan filters out anything that ends with _win32. It's doubtful that this will work properly on Windows.

I am interested in adding mac os support for this library as I have a macbook at hand. Is this project still being maintained?

The Unix version of the crash handler should be improved so that it invokes the previous signal handler after cleaning up, similarly to how the win32 version does it.

I want to classify or customize different event logs to be stored in different files. Does reckless support it?

Hi Mattias!

Thanks for the great library!

Are you aware of the warnings that thread sanitizer reports? They're probably false positives (I hope 😄) but nevertheless they would hide real warnings. Could you please have a look?

Small example:
reckless_tsan.cpp.txt

Build and run like this:

g++ reckless_tsan.cpp -O2 -fsanitize=thread -lreckless && ./a.out

Getting these warnings:
tsan_warnings.txt

After running the benchmark as specified by

https://github.com/Iyengar111/NanoLog/blob/master/nano_vs_spdlog_vs_g3log_vs_reckless.cpp

The code is running successfully with -o0 and -o2 but crashes with -o3.

I'm getting the following crash:

I'm unable to solve the issue for a while.

Please do let me know how to solve this.