opentracing / opentracing-python Goto Github PK

Hi all,
did you already consider to add type hints to your code and to release opentracing as a typed package? It would be great to see that and not even users but also libraries implementing opentracing (e.g. jaeger) would benefit from this.

right now it only tests chaining of set_tag calls

Refer to opentracing.io instead

I'm opening this issue as I'm curious on whether we should adopt the (now merged) OT Java PR for deprecating finishing Spans upon Scope.close().

Problem

For OT Java, finishing Span upon Scope.close() makes it impossible (or hard at least) to report/log errors, as the Span tends to be finished when an Exception is caught. For Python, this is not exactly the same, as Python offers hooks for with statements (the equivalent to try-with Java block) for reporting errors - and as part of #101 we already do that.

So the question is left in case the user wants to report custom errors or additional data upon Exception catching (and how often this truly happens). In OT Java, a close-resources block (try-with statement) can be used to both close the Scope and report the error - for Java, the equivalent block (with statement) does not allow catching Exceptions, and a try-except-finally block is needed instead:

span = tracer.start_span('foo')
scope = tracer.scope_manager.activate(span, False)
try:
  # do work
except Exception as e:
  span.set_tag('...', '...')
  span.log_kv({...})
finally:
  scope.close()
  span.finish()
  # A single call to scope.close() would have been enough, if we had used finish-on-close behavior

Asynchronous frameworks

Most of the time, when working with asynchronous frameworks (gevent, asyncio, tornado) results in calls where we automatically can close the Span and Scope:

with tracer.start_active_span('foo', True):
  # Can close the Span after yielding/awaiting on any coroutine here.
  # Errors would be caught already with the with-block hooks mentioned earlier
  yield my_other_coroutine()

Removing finish-span-on-close means asynchronous frameworks instrumentation will have to handle their Spans lifetime themselves, even if the above case covers the common case:

span = tracer.start_span('foo')
with tracer.start_span('foo') as span:
  with tracer.scope_manager.activate(span): # Remember, not closing Span anymore
    # do things

Or, when reporting errors, a full try statement as described above.

Scope.span being deprecated

As part of the OT Java change, Scope.span access has been deprecated, as we want to prevent the users from passing Scope objects between threads.

However, Tornado instrumentation already is passing Scope objects down the chain between coroutines (the equivalent to threads), even if it's passed in a defacto read-only fashion - similarly, C# with its AsyncLocalScopeManager implementation passes a reference to Scope down the callbacks (which is also how the future contextvars module will work for Python 3.7). Probably it's all-right to do that as long as we do it behind the scenes to not tempt the user to juggle the Scope object himself?

Also, deprecating it here would make the usage of start_active_span() slightly complicated, as we wouldn't have a direct reference to Span:

with tracer.start_active_span('foo') as scope:
   # no scope.span
   tracer.active_span('...', '...')

@yurishkuro @pglombardo @opentracing/opentracing-python-maintainers
cc @indrekj (would be of interest, as we are touching a dynamic language ;) )

but keep the package name as opentracing

It seems like we'd want to do the following if a span is closed exceptionally, given that we're already logging the exception to the span:

def __exit__(self, exc_type, exc_val, exc_tb):
    if exc_type:
        self.set_tag(tags.ERROR, True)
    ...

If others agree I can open a PR

https://github.com/opentracing/opentracing-python/blob/master/opentracing/mocktracer/text_propagator.py#L58 I get an error here when using the MockTracer with my Starlette tests, it's trying to compare bytes with string.

proposed fix:

elif k.startswith(prefix_baggage.encode()):
        baggage[k[len(prefix_baggage):]] = v

Hi all,

Everything has worked fine for ContextVarsScopeManager if I have only Fastapi application. But, When I tried to mount a Flask WSGIApplication to the Fastapi application, I've got an error from ContextVarsScopeManager as "<Token var=<ContextVar name='scope' at 0x7ff47d568a90> at 0x7ff47d61fe00> was created in a different Context". This error has been raised from after_request function of tracer.wrappers.fastapi.fastapi_wrapper module while trying to close scope. I've created a sample project to reproduce the issue and shared it's Github url below.

Github url: https://github.com/gokhan721/opentracing_contextvar_work

Thanks.

In setup.py, we have:

    install_requires=[
        'futures',
    ],

Was it actually meant to be 'future' and not 'futures'?

I don't see concurrent.futures being used anywhere. But I do see _future_ being used in the code.

Per https://github.com/openzipkin/zipkin/blob/master/zipkin-thrift/src/main/thrift/com/twitter/zipkin/zipkinCore.thrift

/**
 * The domain portion of the URL or host header. Ex. "mybucket.s3.amazonaws.com"
 *
 * Used to filter by host as opposed to ip address.
 */
const string HTTP_HOST = "http.host"

/**
 * The HTTP method, or verb, such as "GET" or "POST".
 *
 * Used to filter against an http route.
 */
const string HTTP_METHOD = "http.method"

/**
 * The absolute http path, without any query parameters. Ex. "/objects/abcd-ff"
 *
 * Used to filter against an http route, portably with zipkin v1.
 *
 * In zipkin v1, only equals filters are supported. Dropping query parameters makes the number
 * of distinct URIs less. For example, one can query for the same resource, regardless of signing
 * parameters encoded in the query line. This does not reduce cardinality to a HTTP single route.
 * For example, it is common to express a route as an http URI template like
 * /resource/{resource_id}. In systems where only equals queries are available, searching for
 * http/path=/resource won't match if the actual request was /resource/abcd-ff.
 *
 * Historical note: This was commonly expressed as "http.uri" in zipkin, eventhough it was most
 * often just a path.
 */
const string HTTP_PATH = "http.path"

/**
 * The entire URL, including the scheme, host and query parameters if available. Ex.
 * "https://mybucket.s3.amazonaws.com/objects/abcd-ff?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Algorithm=AWS4-HMAC-SHA256..."
 *
 * Combined with HTTP_METHOD, you can understand the fully-qualified request line.
 *
 * This is optional as it may include private data or be of considerable length.
 */
const string HTTP_URL = "http.url"

/**
 * The HTTP status code, when not in 2xx range. Ex. "503"
 *
 * Used to filter for error status.
 */
const string HTTP_STATUS_CODE = "http.status_code"

/**
 * The size of the non-empty HTTP request body, in bytes. Ex. "16384"
 *
 * Large uploads can exceed limits or contribute directly to latency.
 */
const string HTTP_REQUEST_SIZE = "http.request.size"

/**
 * The size of the non-empty HTTP response body, in bytes. Ex. "16384"
 *
 * Large downloads can exceed limits or contribute directly to latency.
 */
const string HTTP_RESPONSE_SIZE = "http.response.size"

This issue tracks the push for releasing the new 2.0.0 API for OpenTracing-Python.

Based on the Java 0.31 API, the next version of the Python edition of OpenTracing is now available and ready for testing.

RC Branch: https://github.com/opentracing/opentracing-python/tree/v2.0.0

Current Status

• The Scope concept has been brought from the Java API, and is being validated.
• Tracer.start_active_span() defaults finish_on_close=True, after feedback received for the first Release Candidate.
• start_active_span has been defined to have operation_name as a required parameter, as opposed to how it's handled in start_span.
• Properties have been used for new members (such as Scope.span), following the current style (Span.context, for example).
• Imported a testbed suite, which includes a few instrumentation patterns, so API changes and experimental features can be tested out there. This is the equivalent to opentracing-testbed for the Java API.
• Documentation may need to be re-reviewed and extended.
• basictracer-python has been updated as well.

Release process

• The new API is being developed on the v2.0.0 branch.
• Revisions should be requested as Pull Requests against this branch.
• master will continue to point at the latest patch version of the v2.0.0 branch, until 2.0.0 is officially released.
• Changes to the release candidate will be released on PyPi as 2.0.0.rc1, 2.0.0.rc2, etc.

Current RC snapshots

• https://pypi.python.org/pypi/opentracing/2.0.0rc2
• https://pypi.python.org/pypi/basictracer/3.0.0rc2

Hi all,

I use opentracing and would like to reference objects with the help of autodoc and intersphinx. This works by using the objects.inv file. However, the current file doesn't contain full qualified names and therefore referencing object doesn't work. Can this be changed? Also, the documentation should list all classes and not only a few

I've got Zipkin running at http://127.0.0.1:9411

podman run -d -p 127.0.0.1:9411:9411 openzipkin/zipkin

Now how to tell opentracing to send traces there? I don't see any HOST/PORT parameters.

After digging into the opentracing on Python for the last couple of hours it's my understanding that python3 isnt supported. Imho the repo authors should be super-clear about this and call it out in the readme.

@carlosalberto I finally had a chance to look at the new Scope API. It looks fine, but what about the implementations? There was a PR almost a year ago that showed how the API works with different async frameworks like Tornado, gevent, etc.

There were some fixes to the test harness that we need.

The package should probably include the compiled regex that baggage keys should match, especially since opentracing-go provides something similar.

In this test a scope is being opened, but it's left unclosed. This might cause issues, since an active scope will persist in thread local storage even though a new tracer is created, causing another test to fail.

This can be treated as an issue with tests, or a broader one - the key for active scope in thread local storage being same for all tracers created in that thread.

Tornado removed the stack_context module (see tornadoweb/tornado@45d9f6d ) so current test runs are failing with:

==================================== ERRORS ====================================
____________ ERROR collecting tests/scope_managers/test_tornado.py _____________
tests/scope_managers/test_tornado.py:27: in <module>
    from opentracing.scope_managers.tornado import TornadoScopeManager
opentracing/scope_managers/tornado.py:24: in <module>
    import tornado.stack_context
E   ModuleNotFoundError: No module named 'tornado.stack_context'
===================== 104 passed, 1 error in 0.89 seconds ======================
make: *** [test] Error 1
The command "make test testbed lint" exited with 2.

Issue #115 claims to have resolved this error. However, I still get the error with the latest pip packages.
What can I do to chase this down?

pip packages:

# pip3 list | egrep "opent|tornado|jaeger"
jaeger-client               4.3.0
opentracing                 2.3.0
opentracing-instrumentation 3.3.1
tornado                     6.0.3

Traceback (most recent call last):
  File "/av.runfiles/__main__/production/mission_service/mission_service_grpc.py", line 19, in <module>
    from production.mission_service import mission_service_lib
  File "/av.runfiles/__main__/production/mission_service/mission_service_lib.py", line 13, in <module>
    from opentracing_instrumentation import traced_function
  File "/usr/local/lib/python3.5/dist-packages/opentracing_instrumentation/__init__.py", line 21, in <module>
    from .request_context import get_current_span  # noqa
  File "/usr/local/lib/python3.5/dist-packages/opentracing_instrumentation/request_context.py", line 26, in <module>
    from opentracing.scope_managers.tornado import TornadoScopeManager
  File "/usr/local/lib/python3.5/dist-packages/opentracing/scope_managers/tornado.py", line 24, in <module>
    import tornado.stack_context
ImportError: No module named 'tornado.stack_context'
2020-10-19 17:44:02,240 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-19 17:44:03,096 INFO exited: grpc (exit status 1; not expected)
2020-10-19 17:44:04,116 INFO spawned: 'grpc' with pid 141
2020-10-19 17:44:05,139 INFO success: grpc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Traceback (most recent call last):
  File "mission_service.py", line 34, in <module>
    import opentracing_instrumentation.client_hooks
  File "/usr/local/lib/python3.5/dist-packages/opentracing_instrumentation/__init__.py", line 21, in <module>
    from .request_context import get_current_span  # noqa
  File "/usr/local/lib/python3.5/dist-packages/opentracing_instrumentation/request_context.py", line 26, in <module>
    from opentracing.scope_managers.tornado import TornadoScopeManager
  File "/usr/local/lib/python3.5/dist-packages/opentracing/scope_managers/tornado.py", line 24, in <module>
    import tornado.stack_context
ImportError: No module named 'tornado.stack_context'

Don't get me wrong. I love typing.

But I think the current two examples (inbound, outbound) of the README are too long.

I guess more could be done in the library.

Why is it not possible to write the implementation of before_http_request() and before_request() in one line?

If I want to test with MockTracer, and in my business code it calls start_span with child_of parameter, an error will occur as original SpanContext doesn't have _baggage property

        # Assemble the child ctx
        ctx = SpanContext(span_id=self._generate_id())
        if parent_ctx is not None:
>           if parent_ctx._baggage is not None:
E           AttributeError: 'SpanContext' object has no attribute '_baggage'

opentracing.mocktracer.SpanContext has _baggage property, but how can I use it in my business code?

Hello,

The "mock" PyPI package is a backport of Python3.3's unittest.mock for old Python versions.
It is being progressively removed from distributions.
Do you intend opentracing-python to be compatible with Python < 3.3?

If yes, please use "mock" only as fallback:

try:
    from unittest import mock
except ImportError:
    import mock

And in setup.py, require "mock" only in < 3.3:

            'mock;python_version<"3.3"',

If not, you can directly use unittest.mock:

from unittest import mock

And remove the dependency on "mock" in setup.py.

See for example this page which documents the removal of "mock" in
Fedora: https://fedoraproject.org/wiki/Changes/DeprecatePythonMock

Hey folks,

Thanks for this great project. Would it be possible to distribute also as a wheel? This is quite handy when it comes to packaging dependencies afterwards.

This may help:
https://github.com/pypa/cibuildwheel

Best,

Hi!

Why does the MockTracer use hex for incrementing the span and trace ids?

Thanks

I've been making some experiments with supporting Tornado 6 and found that "broken" Tornado scope manager doesn't cause errors in tests.
For example: in this changes condorcet@91ad6c7 I replaced tracer_stack_context with context manager that do noting. Also I removed Tornado specific code from TornadoScopeManager so it just became ThreadLocalScopeManager.
Build with this changes passed successfully https://www.travis-ci.org/condorcet/opentracing-python/builds/513811185
I haven't found tests that modelling situation of switching context in "concurrent coroutines". I think it should be something like this condorcet@30ac1aa

ext.tags in version 1.2.2 is outdated compared to master. Any plans for releasing a new version with updated tags?

The following code, which could have easily been written by mistake, creates an arbitrary number of spans which are never ended and take up enormous amount of memory. Python OpenTracing should have limit the number of open scopes / spans a user can have to prevent a small mistake from becoming an enormous memory leak.

from opentracing.mocktracer import MockTracer

tracer = MockTracer()

while True:
    with tracer.start_span(operation_name='make_some_request') as client_span:
        tracer.scope_manager.activate(client_span, True)
        pass

in order to provide a more correct api, expose the tracing creating and tracer management apis not as a global with a comment to "replace"
but rather as a collection of module scope functions that fetch the right tracer

the global variable is pretty much asking for trouble
see the stdlib logging module and logbook for structural examples

I am currently implementing an opentracing handler for the logging library.
It propagates logging calls to log_kv such that Python logging can be used in opentracing.
When a logging method is called it tries to get the current scope with scope = tracer.scope_manager.active.
When the scope is not None (so when a scope is active) it forwards the logging arguments to log_kv.

I want to publish this piece of code because it might be useful also to others.
Should I create a pull request so that this functionality can be added to this repo or do you prefer me to create my own repo which includes only the opentracing handler for the logging library?

This is a simplified version of the code.

import logging

from opentracing import Tracer

class OpenTracingHandler(logging.Handler):
    def __init__(self, tracer: Tracer):
        super().__init__()

        self._tracer = tracer

    def emit(self, record: logging.LogRecord):
        scope = self._tracer.scope_manager.active

        if scope is None:
            return

        span = scope.span

        span.log_kv({'msg': record.msg})

tracer: Tracer = # initialize the tracer

logger = logging.getLogger('Example')
logger.addHandler(OpenTracingHandler(tracer=tracer))

with tracer.start_active_span('foo') as scope:
    logging.warning('This can be seen in opentracing')

Consider this code

    try:
        carrier = {}
        span.tracer.inject(span, SOME_CUSTOM_FORMAT, carrier)
        do_something(carrier)
    except opentracing.UnsupportedFormatException:
        pass  # it's possible tracer does not support out custom format
    except:
        log.exception('Failed to inject tracing span into headers')
        metric.incCounter()

Without a mandate for the tracers (including the Noop tracer) to return an exception on unknown format there is no way in the code to know if the situation is legitimate (i.e. tracer does not support given custom format) or a genuine error (bad context propagation or what not).

Could you please consider to use unittest.mock at least as fallback?

I have a Python Flask application that uses the requests library. I am trying to implement OpenTracing in the application. A simplified version is:

from flask_opentracing import FlaskTracer
from flask import Flask
from jaeger_client import Config
from opentracing_instrumentation.client_hooks import install_all_patches
from os import getenv
import requests

JAEGER_HOST = getenv('JAEGER_HOST', 'localhost')

def initialise_tracer():
    config = Config(
        config={
            'local_agent': {
                'reporting_host': 'jaeger'
            }
        },
        service_name='opentracing_test',
        validate=True
    )
    return config.initialize_tracer()

app = Flask(__name__)
tracer = FlaskTracer(initialise_tracer, trace_all_requests=True, app=app)
install_all_patches()

@app.route('/foo')
def foo():
    r = requests.get('http://localhost/bar')
    return 'foo : ' + r.text

@app.route('/bar')
def bar():
    return 'bar'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=80)

All endpoints are traced because of giving trace_all_requests to FlaskTracer. All requests made with the requests library are traced because of install_all_patches() from opentracing_instrumentation.

My '/foo' endpoint makes a request to '/bar'. When I access the '/foo' endpoint and view the traces, I can see two separate traces: One containing just '/foo', and a second that contains both the requests GET and the call to '/bar'. I would have expected the spans to propagate through the entire process and therefore I should see a single trace with '/foo' -> GET -> '/bar'.

Do I need to manually set the headers on each of my requests to pass the span? It looks like that is exactly what the requests monkey patch should do...

I've been experimenting using pybind11 to wrap my C++ tracer (https://github.com/isaachier/cpp-client) as a Python tracer. The question I have for this project is whether or not I need to inherit from these classes in order to be opentracing-python compliant. Seeing as Python subclasses are somewhat unnecessary because of Python's duck typing, I'm wondering how far I should pursue subclassing versus just implementing the API without the inheritance.

As far as I can see, the span.set_tag always converts non-string values such as int or bool to string upon storing the value in the tag.

For example if storing a http status code from requests the result tag obj contains a string:

            if r.status_code:
                span.set_tag(tags.HTTP_STATUS_CODE, r.status_code)

I won't even pretend to understand the complexity of the opentracing library, but we have other systems pushing bool and int-type tags into our Jaeger infrastructure, so at least this should theoretically supported. No idea how to start figuring this one out, so any pointers would be appreciated.

Relevant package versions:

jaeger-client==3.11.0
opentracing==1.3.0

1. New context manager ContextVarsScopeManager.
2. Supported Tornado >= 6.

If a before_request hook is registered before the Flask middleware then a span is never setup and the teardown function will log an error when it runs.

from flask import Flask
from opencensus.trace.ext.flask import flask_middleware

app = Flask(__name__)

@app.before_request
def before():
    raise Exception('test')

flask_middleware.FlaskMiddleware(app)

@app.route('/')
def hello_world():
    return 'Hello World'

$ python3 -m venv ./venv
$ ./venv/bin/pip install Flask==1.0.2 opencensus==0.1.10
$ FLASK_APP=sample.py ./venv/bin/flask run
$ curl http://localhost:5000/

Failed to trace request
Traceback (most recent call last):
  File "./venv/lib/python3.7/site-packages/opencensus/trace/ext/flask/flask_middleware.py", line 235, in _teardown_request
    message=str(exception)
AttributeError: 'NoneType' object has no attribute 'status'
127.0.0.1 - - [11/Dec/2018 06:35:05] "GET / HTTP/1.1" 500 -

This occurs because get_current_span() can return None.

if exception is not None:
    span = execution_context.get_current_span()
    span.status = status.Status(
        code=code_pb2.UNKNOWN,
        message=str(exception)
    )

I would have created a PR but I don't know the preferred way to handle this type of issue (conditional around span, warn about span not getting setup, create a new span, etc...).

Hello,

How can we track the db transactions with opentracing tracer or jager tracer.Saw this code and trying to understand how we analyze a db instance within a span with the below variables declared.

Expecting a view like this

https://eng.uber.com/wp-content/uploads/2017/02/8-Screen-Shot-Trace-View-768x391.png

or alternatively we should add a GRPC Format type.

We found a use case where GRPC requires lower-cased headers with the HTTP Propagator: instana/python-sensor#76

I'm still undecided on how to best handle this (see my comments in that PR) but I wanted to get some input on this use case.

Any opinions/ideas?

As it stands, the interfaces do not provide a way to access the trace id, nor the span id.

Jaeger client exposes span.trace_id and span.span_id, but if the tracer is not initialised, that obviously blows up, as the noop span doesn't have those attributes, requiring people to do hasattr checks.

It's useful to log the trace id/span id of the request being started, which can later be used to debug issues with the request somewhere downstream.

We can either emphasize safety by requiring tracer to always take copies of passed collections if it wants to store them, or emphasize performance by telling the caller that it must always give up ownership of collections.

My vote is to emphasize performance. The safety can be achieved by decorators.

The readme states that the default scope manager is thread-local but the reality is that ScopeManager class is used. Either the readme should be changed or the code. IMHO changing the code would be better since using just ScopeManager isn't really useful.

Hello,

I would like to use MockTracer and HTTP_HEADERS, but all I get is a SpanContextCorruptedException

import opentracing
import opentracing.mocktracer

opentracing.tracer = opentracing.mocktracer.MockTracer()
opentracing.tracer.extract(opentracing.propagation.Format.HTTP_HEADERS, {"host": "localhost"})

Traceback (most recent call last):
  File "dumm.py", line 5, in <module>
    opentracing.tracer.extract(opentracing.propagation.Format.HTTP_HEADERS, {"host": "localhost"})
  File "venv/lib/python3.7/site-packages/opentracing/mocktracer/tracer.py", line 185, in extract
    return self._propagators[format].extract(carrier)
  File "venv/lib/python3.7/site-packages/opentracing/mocktracer/text_propagator.py", line 62, in extract
    raise SpanContextCorruptedException()
opentracing.propagation.SpanContextCorruptedException

It seems like the default propagator for HTTP_HEADERS is not suitable for HTTP_HEADERS but only for TEXT_MAP.
Can you please support?

I don't get any change from the carrier when I use tracer.inject(span, Format.BINARY, text_carrier).
therefore I went to see the source code I find maybe it didn't do anything.

I want to ask if I dd something wrong, or is the program not finished?

tracer.py

def inject(self, span_context, format, carrier):
    """Injects `span_context` into `carrier`.
    The type of `carrier` is determined by `format`. See the
    :class:`Format` class/namespace for the built-in OpenTracing formats.
    Implementations *must* raise :exc:`UnsupportedFormatException` if
    `format` is unknown or disallowed.
    :param span_context: the :class:`SpanContext` instance to inject
    :type span_context: SpanContext
    :param format: a python object instance that represents a given
        carrier format. `format` may be of any type, and `format` equality
        is defined by python ``==`` equality.
    :type format: Format
    :param carrier: the format-specific carrier object to inject into
    """
    if format in Tracer._supported_formats:
        return
    raise UnsupportedFormatException(format)