elastic / ecs-logging-python

ecs-logging-python

Check out the Elastic Common Schema (ECS) reference for more information.

The library currently implements ECS 1.6.

Installation

$ python -m pip install ecs-logging

Documentation

See the ECS Logging Python reference on elastic.co to get started.

Elastic APM Log Correlation

ecs-logging-python supports automatically collecting ECS tracing fields from the Elastic APM Python agent in order to correlate logs to spans, transactions and traces in Elastic APM.

License

Apache-2.0

ecs-logging-python's Issues

Transform Elastic APM tracing fields into ECS fields

• Elastic APM logging fields: https://github.com/elastic/apm-agent-python/blob/a898c343bc075e2af5c60ac20ad422cc5290b877/elasticapm/handlers/logging.py#L229
• ECS Tracing Fields: https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html
• span.id should be done as well, will be released in next ECS: elastic/ecs#99

Add an example on how to configure with dictConfig + StdlibFormatter

LOGGING_CONFIG = {
             "format": "[%(asctime)s.%(msecs)03d] %(levelname)s %(name)s:%(funcName)s: %(message)s",
             "datefmt": "%d/%b/%Y:%H:%M:%S",
         },
+        "ecs": {
+            "()": "ecs_logging.StdlibFormatter"
+        },
         "access": {"format": "%(message)s",},
     },
     "handlers": {
         "default": {
             "class": "logging.StreamHandler",
             "stream": "ext://sys.stdout",
-            "formatter": "default",
+            "formatter": "ecs",
         },
         "access_logs": {
             "class": "logging.StreamHandler",

Directly adding fields to json log when log has parameterized string

is something like this possible
example:

addItem='abcd'
logger.info(f"added todo item {addItem} to the todo list")

*note: or any way of adding parametrized string
log:
{'name': 'app.routes.controller', 'msg': 'added todo item abcd to the todo list', 'args': (), 'levelname': 'INFO', 'levelno': 20, 'fields': {'addItem': 'abcd'}}

service.name is set to None because of elasticapm_service_name

Hi,

If you have an application that adds a key like this to its log:

{"service": {"version": 1.0.0, "name": "myapp", "environment": "dev"}}

Then you end up having service.name be set to None and ultimately be deleted from the output, because of this line:

ecs-logging-python/ecs_logging/_stdlib.py

Line 227 in 9f35b52

extras["service.name"] = extras.pop("elasticapm_service_name", None)

Maybe this should instead set service.name to prefer elasticapm_service_name, but fall back to service.name if it already exists and isn't None, i.e.:

extras["service.name"] = extras.pop("elasticapm_service_name", extras.get("service.name", None))

I'm not using Elastic APM and ran into this issue.

Flask example?

It would be great to have an example that works with flask.

[META 27] Align ecs-logging-python

See elastic/ecs-logging#27 for details on alignment.

• Implementation
• Spec validation (#31)
• OSS compliance
• Docs (#29)

stdlib and structlog on separate URLs

I recently advised a developer to add ecs_logging.StructlogFormatter() to our project. However he managed to use the stdlib code because they are on same page.

I propose that stdlib setup and structlog setup should be separate pages in the documentation to avoid this confusion.

https://www.elastic.co/guide/en/ecs-logging/python/current/installation.html#structlog

Also thank you for your great products!

Update ecs.version to match spec

The updated spec expects ecs.version to be dotted, not nested, to match log.level.

how to prefix structlog log value with "custom." or route them to a certain ecs field

Hi,
according to the ecs documentation non ecs standard fields should go under "custom."
Also I might log data that should go to a certain ecs field

but i'm not sure how to configure something like this in structlog/ecs-logging.

for example this:

req_id = uuid()
...
logger.info("something happend", foo="123", request_id=req_id)

should end up being something like this:

{
 "http.request.id" : "<generated_req_uuid>",
 "message": "something happend",
 "custom.foo" : 123
...
}

add additional context with the standard library logging module

Is it possible to add additional context using the standard library logging module?

In other libraries like python-json-logger, there is a concept of an extra field. My goal is to add some additional context like event.id more easily to the logged message without binding to a structlog

StructlogFormatter throws TypeError with duplicated dotted keys

If the event_dict contains a key that is also part of a dotted key the StructlogFormatter throws a TypeError.

Reproducible example:

from ecs_logging import StructlogFormatter

logs = {'event': 'RuntimeError', 'event.dataset': 'dev_setup.log'}

matter = StructlogFormatter()
matter("", "name", logs)

Results in a TypeError

TypeError: 'str' object does not support item assignment

I think the problem is in this line.

def merge_dicts(from_, into):
    """Merge deeply nested dictionary structures.
    When called has side-effects within 'destination'.
    """
    for key, value in from_.items():
        if isinstance(value, dict):
            merge_dicts(value, into.setdefault(key, {}))   <------
        else:
            into[key] = value
    return into

into.setdefault() returns the value of the key if the key is already present in the dictionary. In the example above the value is a string which causes the TypeError.

does not support a dictionary of list of strings

Currently normalize_dict blows up on the following if added the the tests

assert normalize_dict({"a": ["1", "2"]}) == {"a": ["1", "2"]}

Given the type signature it seems like this should be supported. It looks like since normalize_dict is deconstructing any lists with

        elif isinstance(val, list):
            val[:] = [normalize_dict(x) for x in val]

this assumption is that all elements in the list are also dictionaries. It seems like it should be okay to have a list of non dictionary items.

flit_core requirement

Hi,
Is there a reason why ecs-logging-python cannot use flit_core 3.x?
I'm having trouble with depending on flit_core 2.x as it is conflicting with our packages.
Thanks

Make log.level a nested field to satisfy ECS spec

I can see from the code comments that the nested property log.level created within format_to_ecs is then intentionally flattened in order to comply with the ECS schema but I haven't been able to find any reference to this in the ECS documentation, which seems to suggest that it should be a nested field.

This results in output which looks like this:
{ "log.level": "info" }

instead of output which looks like this:
{ "log: { "level": "info" } }

Can anyone point me to the relevant docs?

ECS schema validation in logger

Hi,

This is more of a question than an issue. Is there a validation feature in ecs-logging-python library? I wish to avoid mapping conflict later on during the application on-boarding process, and I wish to make the app ECS compatible from it's very inception.

Currently adding extra fields is simple:

logger.debug(f"Creating directory. {x}", extra={"not_an_ecs_field.method": "get", "source.ip": 22})

I have added two fields, one of which is not really an ECS schema field while the other has an invalid data type. I was wondering if I could pass in the ECS schema as a JSON and have the logger do the validation. I read an option to pass in a validator here, but not sure what it expects.

Linting stage in the CI failed

Linting failed with:

[2021-12-16T17:45:07.296Z] 14 files would be left unchanged.
[2021-12-16T17:45:07.296Z] nox > flake8 --ignore=E501,W503 noxfile.py tests/ ecs_logging/
[2021-12-16T17:45:08.243Z] nox > mypy --strict --show-error-codes --no-warn-unused-ignores ecs_logging/
[2021-12-16T17:45:10.160Z] ecs_logging/_stdlib.py:115: error: Argument "style" to "__init__" of "Formatter" has incompatible type "str"; expected "Union[Literal['%'], Literal['{'], Literal['$']]"  [arg-type]
[2021-12-16T17:45:10.160Z] Found 1 error in 1 file (checked 5 source files)
[2021-12-16T17:45:10.160Z] nox > Command mypy --strict --show-error-codes --no-warn-unused-ignores ecs_logging/ failed with exit code 1
[2021-12-16T17:45:10.160Z] nox > Session lint failed.
script returned exit code 1

Feature parity for structlog

The structlog logger only sets the log level, timestamp and log message while the stdlib logger is setting a lot of more stuff like exceptions, stack infos, thread/process infos. It would be great to have a feature parity here.

Using orjson for serialization

Hi,

While reading Structlog’s performance guide, I wondered how to integrate orjson with ecs_logging.StructlogFormatter. From what I gathered, the standard library’s json module is used in _utils.json_dumps to provide encoding for both formatters. Would you consider a configurable serializer for this project, to gain some performance? I assume the integration would be pretty straightforward, besides some minor changes in json_dumps and a new optional argument for the formatters’ initializers. If you agree, I would be happy to contribute. :)

docs: Add ecs-logging-python to the Elastic doc build

To do

• Move docs to asciidoc #29
• Open PR in elastic/docs to add to the build elastic/docs#2037
• Open infra ticket to add the elasticsearch/ci docs check to this repo https://github.com/elastic/infra/pull/25908
• Add the elasticsearch/ci docs webhook to the repo
• Remove content from readme and link to published docs #30

Ping me when you're ready for this step and I can help you out!

structlog integration - TypeError when passing object

Right now when you add an object to your log like this:

            logger.info(
                "request_started",
                request=request,
                user_agent=request.META.get("HTTP_USER_AGENT"),
            )

You end up with TypeError like this: <WSGIRequest: GET '/healthcheck/'> is not JSON serializable in json_dumps (ecs_logging/_utils.py):

        if value:
            return "{%s,%s" % (
                ordered_json,
                json.dumps(value, sort_keys=True, separators=(",", ":"))[1:],
            )

You should use a default argument for json.dumps and provide fallback mechanism to handle this situation. See https://www.structlog.org/en/stable/_modules/structlog/processors.html#JSONRenderer and _json_fallback_handler.

[Chore] Switch from py2 type comments to py3 type hints

#78 removed py2 support, but didn't switch all the type hinting over. That still needs to be done.

Add Global context fields when creating ecs_logging.StdlibFormatter

I know that based upon the work associated with #7 that context fields can be added to a given logging record, but what I would like to set is a default group that are populated on all log records and then any that are added via the extras on a given record they are added.

So I'd like to be able to do something like:

handler.setFormatter(ecs_logging.StdlibFormatter(extra={"environment": "dev"}))

Related discussion on the nodejs logger here.

StdlibFormatter has an incorrect constructor signature

StdlibFormatter inherits from logging.Formatter, but has a completely different constructor signature, making it incompatible with anything that takes the formatter class name.

StdlibFormatter signature: def __init__(self, stack_trace_limit=None, exclude_fields=()):
logging.Formatter signature: def __init__(self, fmt=None, datefmt=None, style='%', validate=True):

Any time you pass "ecs_logging.StdlibFormatter" where a library takes a class name for a log formatter (Django, Gunicorn, etc.), it fails with the following error:

  File "/usr/local/lib/python3.8/logging/config.py", line 71, in fileConfig
    formatters = _create_formatters(cp)
  File "/usr/local/lib/python3.8/logging/config.py", line 119, in _create_formatters
    f = c(fs, dfs, stl)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given

Add support for stack_info=True to StdlibFormatter

See: https://stackoverflow.com/questions/63052201/logging-formatter-in-python-causes-stack-info-to-be-printed-twice

Add a context manager for temporarily adding extras to StdlibLogger

Similar to how structlog provides .bind() we could provide a context manager that adds extras so you don't have to include them every time you call .info(), etc.

Version 1.1.0 taged is wrong

Hello,

When i'm reading the doc of 1.1.0 i see that the StdlibFormatter takes an argument extra but when i installing the package from pypi (1.1.0) there is no extra argument in __init__ !

Was there a tag problem ?

Thank for your answer,

Best regards,