honeycombio / libhoney-py

Python library for sending data to Honeycomb

libhoney-py

Python library for sending events to Honeycomb, a service for debugging your software in production.

• Usage and Examples

For tracing support and automatic instrumentation of Django, Flask, AWS Lambda, and other frameworks, check out our Beeline for Python.

Contributions

Features, bug fixes and other changes to libhoney are gladly accepted. Please open issues or a pull request with your change. Remember to add your name to the CONTRIBUTORS file!

All contributions will be released under the Apache License 2.0.

Releases

You may need to install the bump2version utility by running pip install bump2version.

To update the version number, do

bump2version [major|minor|patch|release|build]

If you want to release the version publicly, you will need to manually create a tag v<x.y.z> and push it in order to cause CircleCI to automatically push builds to github releases and PyPI.

libhoney-py's Issues

TornadoTransmission: fetch errors are being handled by event loop

Versions

• Python: 3.8.8
• Libhoney: 1.11.1

Steps to reproduce

1. Tornado project, use the TornadoTransmission
2. Generate honeycomb events, but block network access so that sending will fail

The event loop will handle those errors, making it very difficult to diagnose what's happening.

Additional context

On this line: https://github.com/honeycombio/libhoney-py/blob/main/libhoney/transmission.py#L375

        @gen.coroutine
        def _send_batch(self, destination, events):
            ''' Makes a single batch API request with the given list of events. The
            `destination` argument contains the write key, API host and dataset
            name used to build the request.'''
            start = time.time()
            status_code = 0

            try:
                # enforce max_concurrent_batches
                yield self.batch_sem.acquire()
                url = urljoin(urljoin(destination.api_host, "/1/batch/"),
                              destination.dataset)
                payload = []
                for ev in events:
                    event_time = ev.created_at.isoformat()
                    if ev.created_at.tzinfo is None:
                        event_time += "Z"
                    payload.append({
                        "time": event_time,
                        "samplerate": ev.sample_rate,
                        "data": ev.fields()})
                req = HTTPRequest(
                    url,
                    method='POST',
                    headers={
                        "X-Honeycomb-Team": destination.writekey,
                        "Content-Type": "application/json",
                    },
                    body=json.dumps(payload, default=json_default_handler),
                )
                
                ############### THIS LINE ###############
                self.http_client.fetch(req, self._response_callback)
                #########################################
                
                # store the events that were sent so we can process responses later
                # it is important that we delete these eventually, or we'll run into memory issues
                self.batch_data[req] = {"start": start, "events": events}
            except Exception as e:
                # Catch all exceptions and hand them to the responses queue.
                self._enqueue_errors(status_code, e, start, events)
            finally:
                self.batch_sem.release()

fetch returns a Future, but that future isn't being awaited here. (Or yielded, I guess). Because of this, when exceptions are set on that future (around line tornado/httpclient.py:302), the event loop itself ultimately ends up handling them. We sometimes see these kinds of messages in our logs:

Future exception was never retrieved
future: <Future finished exception=HTTP 599: Timeout while connecting>

This, i.e., not await the future returned from fetch, might be intentional? I am not sure.

What I actually want is to implement retries to handle these kinds of errors. So I am intending to make a custom TornadoTransmission subclass that re-implements _send_batch with retry handling. But to do this, I think I will need to actually yield the future, so that I can catch exceptions here (pseudocode):

...
attempts_remaining = 3
while attempts_remaining:
    try:
        yield self.http_client.fetch(req, self._response_callback)
        break
    except HTTPError as e:
        yield sleep(1.0)  # pseudocode
        attempts_remaining -= 1
else:
    raise Exception('Unable to send events to honeycomb servers')
...

So the question is: should that fetch be awaited?

Sometimes libhoney-py raises an SSL exception and crashes the process

File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.5/site-packages/libhoney/transmission.py", line 71, in _sender
self._send(ev)
File "/usr/lib/python3.5/site-packages/libhoney/transmission.py", line 87, in _send
resp = self.session.send(preq)
File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 612, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.5/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:1977)

Add API reference docs

The link for API reference docs was invalid; the link was removed with PR #79 as this file does not exist.

This issue is for consideration of adding API reference and linking it up in the docs.

Libhoney should trim events that break limits instead of dropping them

Hi, We recently had a problem where a third-party library was failing with a ludicrously large exception message (>90kb). This is obviously a problem with the third-party library and should be changed, but it also showed us a problem with our honeycomb observability setup.

The problem is that these exceptions weren't turning up. The events were being silently dropped by honeycomb because they were larger than the API limits (https://docs.honeycomb.io/api/events/#body). This lead to us losing some time on incorrect assumptions based on the honeycomb analysis.

I propose that libhoney should attempt to report events that are too large by one of several options:

• trimming them to fit within the limits
• reporting "event too large" events

I think that either option would be better than silence.

In the mean time, I have worked around the issue with the following presend hook:

def _presend_hook(event_dict):
    PER_STR_LENGTH_LIMIT = 16 * 1024
    ELLIPSIS = '…'

    for k, v in event_dict.items():
        if isinstance(v, str) and len(v) > PER_STR_LENGTH_LIMIT:
            event_dict[k] = f'{v[:PER_STR_LENGTH_LIMIT - len(ELLIPSIS)]}{ELLIPSIS}'

    return event_dict

Add Python version to the user agent header.

After adding tornado as the transmission method to the user agent header in #127, we decided we'd also like to take note of which version of Python is being used.

This will help in providing support and debug information for operational environments as well as give our team signals as to who may be impacted by dropping support for end-of-life language versions.

• Add python version to the user agent header similar to how we've done with libhoney-rb.
• Follow the the user agent guidance at MDN re: format.

To follow other implementations, it should go in this order:
libhoney/v (transmission/v) beeline/v language/v

Consider a higher resolution timer

For python 2.7 we switched

 @contextmanager
    def timer(self, name):
        '''timer is a context for timing (in milliseconds) a function call.

        Example:

            ev = Event()
            with ev.timer("database_dur_ms"):
                do_database_work()

           will add a field (name, duration) indicating how long it took to run
           do_database_work()'''
        start = datetime.datetime.now()
        yield
        duration = datetime.datetime.now() - start
        # report in ms
        self.add_field(name, duration.total_seconds() * 1000)

to

 @contextmanager
    def timer(self, name):
        '''timer is a context for timing (in milliseconds) a function call.

        Example:

            ev = Event()
            with ev.timer("database_dur_ms"):
                do_database_work()

           will add a field (name, duration) indicating how long it took to run
           do_database_work()'''
        start = time.clock()
        yield
        duration = time.clock() - start
        # report in ms
        self.add_field(name, duration * 1000)

in order to get sub millisecond timings.

I'd PR this simple change, except:

Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.clock()
__main__:1: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
3.8311687

I checked https://github.com/benjaminp/six but there are no entries around the time module.

Sampling breaks FileTransmission due to lack of response queue

Currently Event.send() attempts to write events which have been dropped due to sampling to the client's response queue:

libhoney-py/libhoney/client.py

Lines 194 to 197 in 6600f2d

	if self.block_on_response:
	self._responses.put(response)
	else:
	self._responses.put_nowait(response)

This is problematic because FileTransmission doesn't provide a queue:

libhoney-py/libhoney/client.py

Line 77 in 6600f2d

self._responses = self.xmit.get_response_queue()

libhoney-py/libhoney/transmission.py

Lines 494 to 497 in 6600f2d

	def get_response_queue(self):
	'''Not implemented in FileTransmission - you should not attempt to
	inspect the response queue when using this type.'''
	pass

Resulting in a runtime exception:

AttributeError: 'NoneType' object has no attribute 'put_nowait'

ncclient equivalent??

Hi ,

I have used ncclient (netconf client) previously to configure my juniper/dell or cisco devices. I just allows your python script to leverage on a library to get basic get,Edit etc operations on your device. Is libhoney used in the same way?

if so are there any examples of a sample script. To do say a basic get or edit config?

Drop support for Python 2.7

Python 2.7 reached end of life in January 2020. We are planning on dropping support in an upcoming major release.

Clients don't automatically .close() on exit

atexit is currently used to wait for in-flight events to finish for the global client, but if you create your own client you have to explicitly call .close(). This seems inconsistent and was causing me some issues in short-lived scripts that submit events just before exit using a custom client.

Update for E&S

Given a libhoney configured with a v2 api key and without a dataset
When it sends HNY events
Then it should use unknown_dataset as the dataset

When given a legacy key, the behavior remains the same as today.

Notes on trimming whitespace

Received exception related to Python boolean upon close()

libhoney.close()
File "/usr/local/lib/python2.7/site-packages/libhoney/init.py", line 112, in close
_xmit.close()
File "/usr/local/lib/python2.7/site-packages/libhoney/transmission.py", line 114, in close
self.pending.put(None, true, 10)
NameError: global name 'true' is not defined

Python booleans are either "True" or "False".

Exception raised inside Transmission

Details copied from here: honeycombio/beeline-python#32

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/mario/src/project/appenv/lib/python3.7/site-packages/libhoney/transmission.py", line 113, in _sender
    ev = self.pending.get(timeout=self.send_frequency)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mario/src/project/appenv/lib/python3.7/site-packages/libhoney/transmission.py", line 126, in _sender
    pool.submit(self._flush, events)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 151, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

sending threads don't handle all exceptions

Right now we don't guard against threads going away due to an uncaught exception.

what's the python behavior for this anyway? does it bring the entire process down? the thread down? both are bad (the first obviously much worse)

LICENSE lacks filled-in Copyright holder

Heyo,

I would change this in a PR, but I feel like it should be sorted out upstream by you nice folks (though I'm not a lawyer :P).

Thanks,
Porter

Example app is checked in CI

We want to have a useful working example in the repo. To make sure it doesn't get out of date, and to gain more confidence when merging PRs, the example should be exercised in CI.

Transitive dependency on specific versions of urllib3

Versions

• Python: 3.8
• Libhoney: current

Steps to reproduce

1. libhoney relies on requests = "^2.24.0"
2. requests 2.24 relies on urllib3>=1.21.1,<1.27
3. libhoney actually relies on urllib3>=1.26.12 which introduces 'allowed_methods' to urllib3.util.Retry (used in transmission.py)

libhoney should really list all actual dependencies (not rely on transitive) and should specify the versions correctly.

Additional context

Retry once on send timeout

Is your feature request related to a problem? Please describe.

Our ALB currently has a timeout of 60s, so idle connections from libhoney that time out around the 60s mark may be dropped.

Describe the solution you'd like

Retry once if there is an http timeout, similar to what is done in libhoney-go.

Describe alternatives you've considered

Considering increasing the ALB timeout to 65s to avoid common 60s intervals, but libhoney transmission should retry once regardless because this issue could technically occur at any timeout interval.

SDK should work without explicit init()

The Go SDK lets you send events without calling libhoney.Init(), by transparently initializing transmission threads in the background if necessary. This is nice if you exclusively use builders and avoid global state. The Python SDK should do the same.

Buildevents prevents forked builds from running on CI

Versions

• Python: N/A
• Libhoney: main

Steps to reproduce

1. PRs submitted from a fork will fail because we build events is enabled and requires a CircleCI token which is not provided on forked runs.

Additional context

• We have removed build events from other repos because of this limitation.

As seen in #77

Is gzip compression enabled with single event sending?

I see this PR in the 1.80 release https://github.com/honeycombio/libhoney-py/pull/50/files
However, maybe I am not reading the code right but it seems like only _send_batch() is compressing the event with gzip. Will the request body be compressed with just event.send()?

Perhaps just setting gzip_enabled=True (default) when initializing Transmission would compress the body in addition to set the Content-Encoding header?

libhoney prevents scripts from exiting

Hey,

We noticed recently that all of our scripts can't exit. I tracked this down to libhoney.

If you do this in a fresh virtualenv:

pip install libhoney==1.0.11 && echo "import libhoney; libhoney.init(); print('hi')" > test.py && python test.py

The script will hang. It won't respond to ctrl+c. From htop I discovered that SIGINT does nothing; however, SIGKILL, SIGQUIT, and SIGTERM will kill the script.

I diagnosed with hanging_threads and got this stack trace as the blocking culprit:

--------------------Thread 140560820561664--------------------
  File "/usr/lib/python2.7/threading.py", line 525, in __bootstrap
	self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
	self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
	self.__target(*self.__args, **self.__kwargs)
  File "/usr/share/python/cloud-core/local/lib/python2.7/site-packages/libhoney/transmission.py", line 63, in _sender
	ev = self.pending.get()
  File "/usr/lib/python2.7/Queue.py", line 168, in get
	self.not_empty.wait()
  File "/usr/lib/python2.7/threading.py", line 244, in wait
	waiter.acquire()

I get the same results with 1.0.6 and 1.0.11

Mention in the README that folks might wanna use Beeline instead

I'll be happy to make a PR for it, but just stumbled upon beeline, and it looks like a better entrypoint for a lot of folks.

Can't install libhoney from source

Installing libhoney with --no-binary :all: fails due to failing to install poetry.

I initially saw this when running a dockerfile like this:

FROM debian:buster-20200908-slim as base

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3-dev \
    python3-pip \
    gcc \
    g++ \
    && rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install wheel==0.33.6 pip==20.2.3 setuptools==41.6.0
# Libhoney needs to be installed as a wheel since it otherwise crashes when
# trying to install poetry. Add --only-binary libhoney for this to work.
RUN python3 -m pip wheel --no-cache-dir --no-binary :all: libhoney

But it also fails when trying to do a plain pip install libhoney --no-binary :all: in a venv on macOS.

This is the output:

Collecting libhoney
  Downloading libhoney-1.10.0.tar.gz (23 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /<venv-path>/bin/python3 /<venv-path>/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-build-env-zbg9fvr9/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'poetry>=0.12'
       cwd: None
  Complete output (50 lines):
  Collecting poetry>=0.12
    Downloading poetry-1.0.10.tar.gz (168 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /<venv-path>/bin/python3 /<venv-path>/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/tmpq6u32apz
         cwd: /private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-install-j5nvkvpk/poetry
    Complete output (36 lines):
    Traceback (most recent call last):
      File "/<venv-path>/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/<venv-path>/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/<venv-path>/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
        return hook(config_settings)
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-build-env-jae4uub_/overlay/lib/python3.7/site-packages/intreehooks.py", line 53, in get_requires_for_build_wheel
        return self._backend.get_requires_for_build_sdist(config_settings)
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-build-env-jae4uub_/overlay/lib/python3.7/site-packages/intreehooks.py", line 38, in _backend
        obj = self._module_from_dir(modname)
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-build-env-jae4uub_/overlay/lib/python3.7/site-packages/intreehooks.py", line 25, in _module_from_dir
        mod = importlib.import_module(modname)
      File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-install-j5nvkvpk/poetry/poetry/masonry/__init__.py", line 10, in <module>
        from .builder import Builder
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-install-j5nvkvpk/poetry/poetry/masonry/builder.py", line 1, in <module>
        from .builders.complete import CompleteBuilder
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-install-j5nvkvpk/poetry/poetry/masonry/builders/__init__.py", line 1, in <module>
        from .complete import CompleteBuilder
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-install-j5nvkvpk/poetry/poetry/masonry/builders/complete.py", line 6, in <module>
        from poetry.factory import Factory
      File "/private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-install-j5nvkvpk/poetry/poetry/factory.py", line 10, in <module>
        from clikit.api.io.io import IO
    ModuleNotFoundError: No module named 'clikit'
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /<venv-path>/bin/python3 /<venv-path>/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/tmpq6u32apz Check the logs for full command output.
  WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
  You should consider upgrading via the '/<venv-path>/bin/python3 -m pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /<venv-path>/bin/python3 /<venv-path>/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c8/080nx7hn0gn56zlp1ydzp3040000gn/T/pip-build-env-zbg9fvr9/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'poetry>=0.12' Check the logs for full command output.

I'm not familiar with poetry so I'm not sure where exactly the issue lies, but what is confusing to me is that even if I pre-install poetry into the venv (which also installs clikit), it still fails to install libhoney from source afterwards.

Drop support for Python 3.5, 3.6

Python 3.5 reached end of life in September 2020.
Python 3.6 reached end of life in December 2021.

We are planning on dropping support in an upcoming major release.

See Python website for more information on supported versions.

Re-usable Pluggable Django Middleware

A while ago I fiddled with making a Django middleware that makes it easier to add honeycomb to Django projects, basically just moving one of the Django examples into the package, so you can add it to a project using the following:

MIDDLEWARE_CLASSES = (
    # ...
    'libhoney.contrib.django.middleware.HoneyMiddleware',
)

Would it be worth making a PR or a separate package for this? I can write tests and clean up the required and optional/dynamic fields then too.

django-prometheus and Sentry include similar modules.

https://github.com/honeycombio/libhoney-py/compare/master...zoidbergwill:django-middleware?expand=1

P.S.: Sorry if I missed an issue discussing this.

When using Tornado transmission, add Tornado in the User Agent Header

We'd like to instrument which method of transmission libhoney is using to send batches to the API
Let's add it. ✨

Reference: User-Agent HTTP Headers

Libhoney Ruby does it like this.
It would likely be added around here

Python SDK returning errors when sending events

The python SDK is returning errors when I run the sample code to add an event:

{'body': '', 'status_code': 0, 'duration': 0.1239776611328125, 'metadata': None, 'error': TypeError("request() got an unexpected keyword argument 'json'",)}

The issue seems to in transmission.py, here:
resp = self.session.post(
url,
headers={"X-Honeycomb-Team": destination.writekey},
json=payload)

Changed this to:

       resp = self.session.post(
            url,
            headers={"X-Honeycomb-Team": destination.writekey},
            data=json.dumps(payload))

and it appears to be working.