Add traces with different levels about connxr

[nopeslide]

@alrevuelta
I need to rewrite the conv operator, since it produces wrong data in a network of mine.
Since this operator is large, I would rewrite the logging first to see what is happening inside.
My proposal:

• all tracing/logging is done via macros, so we keep the binary clean if unneeded
• file descriptor is settable for each severity
  • i.e. TRACE_INFO_FD
• tracing can be overwritten per file
  • format: `TRACE_FILE="file[:level]{;file[:level]}"
  • i.e. TRACE_FILE="inference.c:2;main.c:1"
• tracing can be overwritten per function
  • format: `TRACE_FUNCTION="file[:level]{;file[:level]}"
  • i.e. TRACE_FUNCTION="operator_xy:2;inference:1"
• tracing is extended with node/attribute/tensor exports
• exporting can be set globally/ per file/ per function
  • via TRACE_EXPORT, TRACE_EXPORT_FILE, TRACE_EXPORT_FUNCTION
• single generic trace macro with fprintf style arguments
  • TRACE(level, file, format_string, ...)
• various variants that use this generic trace macro
  • severity
    • TRACE_INFO(format_string, ...)
      • file, line, function
      • on TRACE_LEVEL >= 1
      • stdout
    • TRACE_DETAIL(format_string, ...)
      • file, line, function
      • on TRACE_LEVEL >= 2
      • stdout
    • TRACE_DEBUG(format_string, ...)
      • file, line, function
      • on TRACE_LEVEL >= 3
      • stdout
    • TRACE_WARN(format_string, ...)
      • file, line, function
      • on TRACE_LEVEL >= 1
      • stderr
    • TRACE_ERROR(format_string, ...)
      • file, line, function
      • on TRACE_LEVEL >= 1
      • stderr
      • flush
    • TRACE_FATAL(format_string, ...)
      • file, line, function
      • on TRACE_LEVEL >= 1
      • stderr
      • flush
      • abort (generates core dump)
  • data types
    • TRACE_TENSOR(tensor)
      • name, shape, datatype on TRACE_LEVEL >= 1
      • values on TRACE_LEVEL >= 2
      • complete dump if TRACE_LEVEL >= 3
    • TRACE_ATTRIBUTE(attribute)
      • name, datatype on TRACE_LEVEL >= 1
      • values on TRACE_LEVEL >= 2
      • complete dump on TRACE_LEVEL >= 3
    • TRACE_NODE(node)
      • name, op_type, domain, inputs, outputs, attributes on TRACE_LEVEL >= 1
      • values on TRACE_LEVEL >= 2
      • complete dump on TRACE_LEVEL >= 3
  • misc
    • TRACE_ENTER(message)
      • file, line, function
    • TRACE_EXIT(message)
      • file, line, function
    • TRACE_CALL(func_pointer, ...)
      • file, line, function, args, return value
  • conditionals
    • all macros have a conditional variant
      • i.e. TRACE_INFO_IF(condition, format_str, ...)

[alrevuelta]

I need to rewrite the conv operator, since it produces wrong data in a network of mine.

Is it with mobilenetv2? I was playing with the onnx one and seems to fail because the group attribute is not implemented. There are many hardcodings so I wouldn't be surprised if its failing somewhere else. Respect if you manage to fully implement it.
I wrote something recently that can come in handy for debugging see this. Its a new feature that allows to dump all intermediate outputs to a file, and then assert the output each node individually to see which node is causing problems. Quick prototype though.

Regarding the tracing levels.

all tracing/logging is done via macros, so we keep the binary clean if unneeded
Totally agree.

Generally I like the idea but I would simplify some things:

• I don't see the need of the traces under misc and conditionals
• warn, error, and fatal ok.
• info, detail, debug. Why all 3? Do we really need that granularity?
• I liked the current idea of having TRACE_LEVEL0, TRACE_LEVEL1 and TRACE_LEVEL2.

So my idea would be something like:

• Have TRACE_LEVEL0, TRACE_LEVEL1 and TRACE_LEVEL2 as the only macros that call printf.
• Then have other macros that include the previous ones such as:
  • TRACE_WARN
  • TRACE_ERROR
  • TRACE_FATAL
  • TRACE_TENSOR
  • TRACE_ATTRIBUTE
  • TRACE_NODE

So when we want to trace something we can either call the direct TRACE_LEVELX function, or "call" another macro that offers some kind of extra functionality (i.e. TRACE_TENSOR that iterates the tensor and prints different information according to the trace level that is set).

I'm trying to simplify it because I don't want to end up with traces that we don't use, or that are used wrong. The less traces, the less we have to enforce that they are used correctly, because they would be easier to understand.

[nopeslide]

Is it with mobilenetv2? I was playing with the onnx one and seems to fail because the group attribute is not implemented. There are many hardcodings so I wouldn't be surprised if its failing somewhere else. Respect if you manage to fully implement it.

found this bug with my mobilenetv2 network: conv produces wrong shapes (one dimension is off by one), but didn't look into it.
I'm already preparing for the nightmares regarding this operator :D

I'm trying to simplify it because I don't want to end up with traces that we don't use, or that are used wrong.

But by explicitly not stating what this trace_level is for, you are inviting misusage, don't you?
I just renamed your 0 to info, 1 to detail and 2 to debug and incremented them by one since TRACE == 0 should be no trace, shouldn't it?

the misc thing was for entering functions and leaving them, instead of having a funny trace call with a parameter equal to the __func__ macro value.

void operator_xy() {
  TRACE_ENTER(); //start of scope
  ...
  TRACE_EXIT(); //end of scope
  return 0;
}

the conditionals are based on the idea that optional objects may have dependencies.
these macros allow to specify dependencies without writing weird if constructs which only apply to trace calls.

TRACE_ATTRIBUTE(NULL); //should be fatal, since we want to trace sth that does not exist
TRACE_ATTRIBUTE_IF(attribute,attribute);  //attribute is optional
TRACE_ATTRIBUTE_IF(optional_tensor,optional_attribute);  //attribute is needed when tensor is present

they act as a kind of assert, but with integrated logging

[nopeslide]

I wrote something recently that can come in handy for debugging see this.

this is nice, will look into it.
exported pb files can be loaded with python via the onnx library.
I also wrote a utility function to generate these exports so I could compare them with the results of other frameworks.

[alrevuelta]

I just renamed your 0 to info, 1 to detail and 2 to debug and incremented them by one since TRACE == 0 should be no trace, shouldn't it?

Ah okay, didn't saw that. Don't you prefer having 0, 1, 2 or 1, 2, 3? At least to me it looks more straight forward and I don't have to think, is detail more verbose than debug? Or it was the other way?

Regarding TRACE_ENTER and TRACE_EXIT. Why not just using debug or detail? Nothing really against, just trying to save some macros.

the conditionals are based on the idea that optional objects may have dependencies.
these macros allow to specify dependencies without writing weird if constructs which only apply to trace calls.

I'm afraid I don't understand the conditional macros. Could you elaborate?

[nopeslide]

Ah okay, didn't saw that. Don't you prefer having 0, 1, 2 or 1, 2, 3? At least to me it looks more straight forward and I don't have to think, is detail more verbose than debug? Or it was the other way?

fair enough :D
I would prefer names, at least in addition to numbers.
but you are right, debug may not have been the best choice.

Regarding TRACE_ENTER and TRACE_EXIT. Why not just using debug or detail? Nothing really against, just trying to save some macros.

The reasoning behind these was to minimize boilerplate
instead of
TRACE_INFO("enter function") and TRACE_INFO("exit function")
just
TRACE_ENTER() and TRACE_EXIT()
or TRACE_START() and TRACE_STOP()
or TRACE_BEGIN() and TRACE_END()
this enforces a log format, making log searches easier.

I'm afraid I don't understand the conditional macros. Could you elaborate?

of course.
sometimes it's way more compact to have a conditional print.
for example, you have an optional attribute that overrides some default value, you can easily log this via

TRACE_INFO_IF(!attribute,"using default value %d\n",default_value);

instead of specifying an if/else case just for this.
this mechanism also allows for asserts:

TRACE_FATAL_IF(!attribute,"attribute not specified!\n");

and if we say that TRACE_{ATTRIBUTE,TENSOR,...} are fatal if they receive a NULL pointer
we can do the same thing even more compact:

TRACE_ATTRIBUTE(attribute)

would suffice.
if an attribute is optional we could always go back to a conditional variant:

TRACE_ATTRIBUTE_IF(attribute,attribute)

does not fail if attribute is not valid.
if we have an attribute bar that requires foo we can simply express this as

TRACE_ATTRIBUTE_IF(foo,bar)

[nopeslide]

The reasoning behind these was to minimize boilerplate
instead of
TRACE_INFO("enter function") and TRACE_INFO("exit function")
just
TRACE_ENTER() and TRACE_EXIT()
or TRACE_START() and TRACE_END()

another option would be to use any supplied message as a log prefix.
for example (as some kind of csv output):

TRACE_INFO("test");
// info:example.c:34::test
TRACE_START("block xy");
TRACE_INFO("test2");
// info:example.c:34:block xy:test
...

[alrevuelta]

fair enough :D
I would prefer names, at least in addition to numbers.
but you are right, debug may not have been the best choice.

Nothing against but I still prefer numbers. It improves readability. But your choice :)

Lets keep discussing it once we have a PR in front.