census-instrumentation / opencensus-python Goto Github PK

• Implement veneer client library on top of the autogen in google-cloud-python repo. (tracked here)
• Implement Stackdriver Trace V2 Exporter.

labels_helper contains GCP specific labels, we should expose these from not from the core library but from a GCP-specific sub package.

For resolving #63. Currently the spans are stored in spans_list and sent to exporter all together when calling tracer.finish(). Need to change the implementation to export span once the span is ended to support the batch exporting.

(Somewhat related to #105)

I would like to propose that some of the span properties (like start_time/end_time, span_id/trace_id) are stored in a more canonical format (eg, raw datetime.datetime and uuid.UUID respectively) inside the Span object. We can then defer the formatting of this value to whichever exporter is being used.

For example, right now start_time undergoes the following transformations when using the ZipkinExporter:

• When it's created: datetime.datetime -> str
• When it's exported: str -> datetime.datetime -> int

If we stored it as a raw datetime.datetime object, we could simply convert it to an int when reporting and subtract. It would be simple to add the conversion to ISO format when reporting for StackDriver as well. I'm sure the core classes were designed with StackDriver as the first class citizen, but some changes like this would make for easier addition of other Exporters.

Another possible solution is to store the start_time and end_time separately from the duration. This could allow us to use something higher res like time.perf_counter() (at least in Python3) for calculating the duration.

the version of the opencensus package installed using

pip install opencensus

in python3.5 does currently not match the version in git and the examples given fail with errors that python cannot find "Tracer()".

The package is currently being distributed per aspect, e.g. pip install opencensus-trace. For other languages, our policy is to distribute trace, tags and stats in the same package. There are two reasons we are considering this option:

i. Trace, tags and stats are very thin and bundling them together is not costly.
ii. OpenCensus is a unified tracing, stats and propagation library and giving user the opportunity to install a single package is aligned with its unified nature.

Hence, I am suggesting we should distribute the package as "opencensus" with trace, stats, and tags.

Currently, span objects are converted to an intermediate json format before being sent to an exporter. As a result, there are some stackdriver specific conversions being applied to all spans (such as string truncation) that may not necessarily apply to all export formats.

I am proposing to update the base.Exporter emit and export methods to take a raw span.Span object instead of a dict. Not only will this push stackdriver specific details to the stackdriver exporter, but it would also make the development of new exporters easier since there are less layers of conversion.

Happy to make a PR if everyone is ok with the change.

  File "main.py", line 19, in <module>
    from opencensus.trace.tracer import Tracer
ImportError: cannot import name 'Tracer' 

More Details

Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from opencensus.trace import execution_context
>>> from opencensus.trace.tracer import Tracer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Tracer'

Increase batch size and do not wait between each export to ensure all spans are sent within the grace period when terminating the thread.

Following the instructions for integrating requests results in the following error:

WARNING:opencensus.trace.config_integration:Failed to integrate module: requests, supported integrations are httplib, mysql, postgresql, pymysql, requests, sqlalchemy

This is because the docs say request but it should be httplib

There is a bug in OpenCensusServerInterceptor that causes server spans to be finished immediately. You can reproduce it by running the helloworld grpc example in examples/trace/grpc. You should see a server span with a very short execution time, for example when I run it locally with a print exporter I get:

{'traceId': 'a8fdeb7813144b178cb4b3d3d9b8fee4', 'spans': [{'childSpanCount': 0, 'displayName': {'truncated_byte_count': 0, 'value': 'grpc_server'}, 'parentSpanId': 4417962580138865, 'startTime': '2018-01-23T16:01:17.826761Z', 'attributes': {'attributeMap': {'/component': {'string_value': {'truncated_byte_count': 0, 'value': 'grpc'}}}}, 'spanId': 7098108131070600, 'endTime': '2018-01-23T16:01:17.826771Z'}]}

The problem is that this line is being traced, but it isn't actually executing the server handler code. It just returns an RPCMethodHandler, which defers execution.

I hacked up a quick fix that wraps the RPCMethodHandler and instruments the span when the handler is evaluated. A better solution might be to have the RpcMethodHandlerWrapper accept arbitrary callbacks, or to update grpc to have RPCMethodHandler accept callbacks natively.

Happy to submit a PR if you have a preferred approach. Thanks!

And also create a SpanData class.

While writing some new tests, I hit test pollution (i.e. pytest passes with just my new test suite, but fails when running with others), which I think is from the httplib tests (added in #95) not clearing the execution context in the tearDown method.

Here's a snippet from my postmortem pdb session from pytest (from within the test I am writing that fails due to pollution):

        span_id = 1234
        tracer = execution_context.get_opencensus_tracer()
        self.assertIsNotNone(tracer)

        span = tracer.current_span()

>       self.assertEqual(span.parent_span.span_id, span_id)
E       AssertionError: <Mock name='mock.span_id' id='4596810808'> != 1234

tests/unit/trace/ext/pyramid/test_pyramid_middleware.py:233: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/case.py(822)_baseAssertEqual()
-> raise self.failureException(msg)
(Pdb) up
> /usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/case.py(829)assertEqual()
-> assertion_func(first, second, msg=msg)
(Pdb) up
> /Users/plotnick/opencensus-python/tests/unit/trace/ext/pyramid/test_pyramid_middleware.py(233)test__after_request()
-> self.assertEqual(span.parent_span.span_id, span_id)
(Pdb) p execution_context._thread_local.attrs
{'httplib/current_span_id': <Mock name='mock.span_id' id='4596869664'>}

This does not occur after adding execution_context.clear() to the setUp() test suite method (I already had it in the tearDown() method)

The link on the README results in a 404!

https://opencensus.io/opencensus-python/trace/usage.html

This should be

https://census-instrumentation.github.io/opencensus-python/trace/usage.html

With the latest release, 0.1.3, I am getting the following exception when using the google_cloud_clientlibs trace integration:

Exception in thread Thread-19:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 701, in channel_spin
    completed_call = event.tag(event)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 174, in handle_event
    callback()
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 313, in <lambda>
    self._state.callbacks.append(lambda: fn(self))
  File "/usr/local/lib/python2.7/dist-packages/opencensus/trace/ext/grpc/client_interceptor.py", line 122, in callback
    self._tracer.end_span()
  File "/usr/local/lib/python2.7/dist-packages/opencensus/trace/tracer.py", line 116, in end_span
    self.tracer.end_span()
  File "/usr/local/lib/python2.7/dist-packages/opencensus/trace/tracers/context_tracer.py", line 112, in end_span
    self.exporter.export(span_json)
  File "/usr/local/lib/python2.7/dist-packages/opencensus/trace/exporters/stackdriver_exporter.py", line 145, in export
    self.transport.export(trace)
  File "/usr/local/lib/python2.7/dist-packages/opencensus/trace/exporters/transports/sync.py", line 23, in export
    self.exporter.emit(trace)
  File "/usr/local/lib/python2.7/dist-packages/opencensus/trace/exporters/stackdriver_exporter.py", line 142, in emit
    self.client.batch_write_spans(name, stackdriver_spans)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/trace/client.py", line 104, in batch_write_spans
    timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/trace/_gax.py", line 91, in batch_write_spans
    timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/trace_v2/gapic/trace_service_client.py", line 187, in batch_write_spans
    self._batch_write_spans(request, retry=retry, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/google/api_core/grpc_helpers.py", line 54, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_interceptor.py", line 141, in __call__
    return call_future.result()
  File "/usr/local/lib/python2.7/dist-packages/grpc/_interceptor.py", line 110, in result
    raise self._exception
AttributeError: 'NoopTracer' object has no attribute 'span_context'

Here is my code:

    exporter = stackdriver_exporter.StackdriverExporter(
        project_id=project)
    # exporter = print_exporter.PrintExporter()
    spanCtx = tracer_module.SpanContext(trace_id=str(traceId))
    tracer = tracer_module.Tracer(exporter=exporter, span_context=spanCtx)
    config_integration.trace_integrations(['google_cloud_clientlibs'])

• Please add a file called AUTHORS with one line for the moment Google Inc.
• Change all licenses headers to: Copyright [YEAR], OpenCensus Authors

See census-instrumentation/opencensus-java#539

This library only exports spans, trace should not be used internally.

I get the following error when using the Zipkin exporter:

ERROR:root:maximum recursion depth exceeded in cmp
ERROR:root:maximum recursion depth exceeded in cmp
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded in cmp
ERROR:root:maximum recursion depth exceeded in cmp
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded in __instancecheck__
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded in cmp
ERROR:root:maximum recursion depth exceeded in cmp
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded while calling a Python object
ERROR:root:maximum recursion depth exceeded
ERROR:root:maximum recursion depth exceeded

Here is my setup code:

    zipkin_host = os.environ.get('ZIPKIN_PORT_9411_TCP_ADDR')
    exporter = zipkin_exporter.ZipkinExporter(
        host_name=zipkin_host)
    spanCtx = tracer_module.SpanContext(trace_id=str(traceId))
    tracer = tracer_module.Tracer(exporter=exporter, span_context=spanCtx)
    config_integration.trace_integrations(['google_cloud_clientlibs'])

Hello!

I'm researching using opencensus-python as a means to add tracing to applications I'm helping to develop.

One issue I noticed pretty quickly is that in the "translate_to_zipkin" method in the ZipkinExporter class, the start timestamp and durations variables exported to zipkin are rounded to the nearest second, then converted to microsecond units by multiplying by 1,000,000. While the start and end timestamps are converted to microsecond units, they always have 6 trailing zeros regardless of the start and end date times. Here is the code that's doing this -- noting the start_timestamp_ms and end_timestamp_ms:

   for span in spans:
       # Timestamp in zipkin spans is int of microseconds.
       start_datetime = datetime.datetime.strptime(
           span.get('startTime'),
           ISO_DATETIME_REGEX)
       start_timestamp_ms = calendar.timegm(
           start_datetime.timetuple()) * 1000 * 1000

       end_datetime = datetime.datetime.strptime(
           span.get('endTime'),
           ISO_DATETIME_REGEX)
       end_timestamp_ms = calendar.timegm(
           end_datetime.timetuple()) * 1000 * 1000

       duration_ms = end_timestamp_ms - start_timestamp_ms

Python datetime objects can have microsecond precision.
Zipkin metrics can also handle microsecond precision.

To utilize the microsecond precision I wanted, I had to derive my own version of the ZipkinExporter class to override this method. So I calculated the duration as such:

   for span in spans:
       # Timestamp in zipkin spans is int of microseconds.
      start_datetime = datetime.datetime.strptime(
           span.get('startTime'),
           ISO_DATETIME_REGEX)

       end_datetime = datetime.datetime.strptime(
           span.get('endTime'),
           ISO_DATETIME_REGEX)

       duration_ms = int((end_datetime - start_datetime).total_seconds() * 1000 * 1000)

The conversion I changed still involves multiplying by 1,000,000, but it's multiplied to a seconds float value instead of a rounded int as above.

As for setting the start timestamp, in the class I created, I have an epoch datetime constant like this. Since the current code assumes UTC date times, I'm assuming the same:

epoch = datetime.datetime.utcfromtimestamp(0)

..and for converting the start_datetime variable to epoch microseconds, I do this:

'timestamp': int((start_datetime - self.epoch).total_seconds() * 1000 * 1000)

(The above conversion of a timedelta to epoch microseconds can be easily wrapped into a small static method.)

The precision of both the start timestamp and the duration in the class I created now utilizes true microsecond precision with these changes.

Is there a good reason the current code for ZipkinExporter needs to round to whole seconds before exporting?

• Brian

The value should be:
"opencensus-java [" + $VERSION + "]";

1 byte bitmap where the less significant bit means enabled, and the other bits will mean other options in the future.

Currently there is no way for users to set the transport type(sync or async) in Django settings, we need to provide support for them to choose the transport type.

from opencensus.trace.exporters import file_exporter
from opencensus.trace.tracer import context_tracer (not tracers)

exporter = file_exporter.FileExporter(file_name='/tmp/traces')
tracer = context_tracer.ContextTracer(exporter=exporter)

lib/python2.7/site-packages/opencensus/trace/exporters/file_exporter.py

   with open(self.file_name, 'w+') as file:
        trace_str = json.dumps(trace)
        file.write(trace_str)

i change to :

    with open(self.file_name, 'a+') as file:
        trace_str = json.dumps(trace)
        file.write(trace_str)
        file.write('\n')

The product's name is Stackdriver, let's keep it in the name consistently every where.

Integrated with the cloud client libraries by tracing the gRPC and HTTP requests made by the client library calls.

The GoogleCloudExporter should batching multiple traces and upload them to the backend once a while. The batching options could be configurable.

See the bundler utility in Go for an example list of batching/buffering options: https://godoc.org/google.golang.org/api/support/bundler#pkg-constants.

See census-instrumentation/opencensus-java#550 for encoding format.

It is giving users extensive number of options to provide both exporters. Either provide logging or printer exporter. Also, we don't expect users to depend on them other than the development time.

It doesn't seem like stack traces are being set anywhere in the current codebase. It would be great if there was support to automatically attach them to the span when an exception is raised.

Some examples would be:

1. When the span context manager closes in an error state: here
2. When the gRPC server interceptor sees an exception: here
3. When the Flask middleware sees an exception: here (although for this one you might need to add teardown_request).

I have a local branch that implements attaching and exporting stack traces for gRPC spans. I can make a PR if everyone is onboard with the proposal.

We need to support tracing Grpc requests automatically.

• Implement interceptors for client and server
• Add hello world example on how to use the interceptors
• Update README documentation

Django 1.10 added a new middleware interface in settings.py called MIDDLWARE instead of MIDDLEWARE_CLASSES. Current implementation only supports the MIDDLEWARE_CLASSES interface. We need to make the middleware compatible with both interfaces. Django provides MiddlewareMixin to solve this problem.

Use case: On GAE Flex, the default setting for health check is on, and user's application will receive a lot of health check requests which would be automatically traced and thus the Trace UI will show a lot of health check spans. We need a way to enable the users to config what path they do not want to trace.

The Zipkin exporter is currently setting the destination Zipkin server's hostname as the ipv4 on spans here. It should instead be using the hostname/ip of the local host. The python jaeger client uses a good strategy for determining the local ip/hostname in utils.py

https://github.com/TraceContext/tracecontext-spec/blob/master/trace_context/HTTP_HEADER_FORMAT.md

Hello, I'd be very keen to see a prometheus exporter for opencensus-python. I'd like to use this instead of the flask_prometheus which I currently use.

Hi, I started using this for some projects yesterday and definitely love it. I needed to make a few fixes (which I'll be submitting PRs for this weekend), but this definitely fits into it. I had 2 ideas I wanted to run by you before embarking on them.

I think it would make sense to separate the formatting from the exporting. That would allow flexibility to say "format this as Zipkin, but print to the screen/file". We could make this transparent to the user by each exporter having a default formatter and also ensuring compatibility (eg, the StackDriver exporter can only use the StackDriver formatter).

Down the road, this could allow a much simpler pattern for different endpoints to be totally separate subpackages/plugins for each exporter (eg, you would explicitly pip install opencensus-python[stackdriver]).

Need to automatically figure out what trace format the user is using, including Zipkin, AWS, StackDriver and TraceContext headers.

In order languages, it is called probability sampler. See https://github.com/census-instrumentation/opencensus-java/blob/master/api/src/main/java/io/opencensus/trace/samplers/ProbabilitySampler.java as an example.