I've looked at what we've added into the repo for debug logging. The recommendation of how we expect developers to use the logging features was not at all clear. Like everyone else, I can read code and figure this out, but was looking for the "new developers guide to logging". We don't have that yet.
It raises the question of whether we even have agreement and understanding of what we need. Possibly what we've added, is not the right solution.
So, lets discuss requirements and use cases.
The style of debug logging we have in Peloton, is widely used. Many implementations pair the LOG calls with multiple back end implementations, so you can log to stdout, files or network sockets. We use it only for stdout, so it is a more structured way of doing printfs. So, what is wrong with what we have in Peloton?
- We have not applied consistency to what we log and when we log it.
- At LOG_INFO, we don't log much, but some of what is logged is not useful.
- At LOG_DEBUG it is a free for all. Some log points provide useful information, others don't. Often the log information is only useful to the developer who implemented it or needs additional knowledge of the code path to make any sense of the output.
- At LOG_TRACE, the volume of output is so high, it is IMO practically useless. LOG_TRACE is inconsistently used to trace function calls. There are better ways of doing this.
- There is no way to reduce the noise level and zero in on the component you are developing or debugging. You get all LOG_DEBUG messages or none.
Questions
- What do we want to get from debug logging?
- What set of features do we want?
- What controls / knobs do we want?
Opinions
- Debug logging as a structured way of printing information is OK.
- Having multiple back ends would be nice, but is not required. Just printing is usable.
- We need guidelines on what is appropriate in the different log levels.
- We need guidelines on what to put in log messages, and enforce this in code reviews.
- In addition to the log level to be output, we need to be able to control logging "per module". This could be:
- Per "module", e.g. per namespace
- Per module, where it is some other module notion that we encode
- Per file.
- Compile time selection of what is logged, is usable. Run time enable / disable of module logging is desired, possibly required.
- Control of logging via API, at run time, would be very useful. When debugging, this would allow logging to turn on only when some interesting condition or state is entered.
- Control of logging via GDB is useful. Control via some admin user interface is better, but is unlikely in the near future.
- You should be able to completely eliminate overhead when not using it. The Peloton logging does this by providing an empty macro body when you turn off a log level.
- Gcc can compile code that includes hooks for function entry/exit, according to the documentation. Function tracing should use this facility as a compile variant, rather than us manually inserting function entry / exit LOG points.
I was hoping that easylogging would provide a suitably lightweight framework, which could be wrapped to provide a small, controlled subset that fit our needs. I can't tell if it does or not.
Alternatively, if we were rolling our own, a possible approach is as follows. It is very basic and very minimal, and just slightly extends what we've been using.
Outline
-
Start with the Peloton logging macros, i.e. keep the notion of:
- LOG_INFO
- LOG_DEBUG
levels etc.
-
Document what is appropriate for each level.
-
Document and enforce what goes into log messages. The purpose here is to avoid, useless log points being inserted.
-
Extend the macros to include a module. For the moment lets assume modules map roughly to a namespace, but possibly finer granularity, defined by us.
-
A log point might look like:
LOG_DEBUG(MODULE_NETWORK_MARSHALLING, "output binary %d", var);
-
If we could live with 64 modules max, the modules could be encoded as bits, e.g.
#define MODULE_NETWORK_MARSHALLING 0x1
#define MODULE_NETWORK_PARSING 0x2
and so on
If we want more than 64 modules, then you do a lookup rather than a bit mask, so marginally
more complex.
-
Log macros would then be roughly:
if ( log_enabled_var & MODULE_XXX) then LOG_DEBUG_NO_MODULE(string, varags)
-
Couple with some additional macros and / or APIs to control logging at runtime.
Now, this is the familiar approach. The disadvantage to this is printing to stdout is not cheap. It slows down the system significantly and therefore changes timing. Problems that occur at high load when running at high performance, sometimes (frequently?) can't be debugged this way.
Therefore it may be necessary, at some point, to supplement debug logging with "high performance tracing". It may be a long way in the future. I note here just to raise awareness that debug logging is not the end of the story.
This is an old technique for debugging devices drivers, kernel modules etc. The outline being:
- Create a "large" memory buffer. Used as a circular buffer.
- Insert log points that insert items into the memory buffer. Typically binary information only. Log point identifier, values to save for the log point.
- When needed, after some event of interest, programmatically dump the buffer.
- Use associated utility to decode the contents of the buffer.
- This is of course, extremely fast but not so easy to use.
Back to debug logging. Comments?
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent