trallnag / prometheus-fastapi-instrumentator

Instrument your FastAPI with Prometheus metrics.

Prometheus FastAPI Instrumentator

A configurable and modular Prometheus Instrumentator for your FastAPI. Install prometheus-fastapi-instrumentator from PyPI. Here is the fast track to get started with a pre-configured instrumentator. Import the instrumentator class:

from prometheus_fastapi_instrumentator import Instrumentator

Instrument your app with default metrics and expose the metrics:

Instrumentator().instrument(app).expose(app)

Depending on your code you might have to use the following instead:

instrumentator = Instrumentator().instrument(app)

@app.on_event("startup")
async def _startup():
    instrumentator.expose(app)

With this, your FastAPI is instrumented and metrics are ready to be scraped. The defaults give you:

• Counter http_requests_total with handler, status and method. Total number of requests.
• Summary http_request_size_bytes with handler. Added up total of the content lengths of all incoming requests.
• Summary http_response_size_bytes with handler. Added up total of the content lengths of all outgoing responses.
• Histogram http_request_duration_seconds with handler and method. Only a few buckets to keep cardinality low.
• Histogram http_request_duration_highr_seconds without any labels. Large number of buckets (>20).

In addition, following behavior is active:

• Status codes are grouped into 2xx, 3xx and so on.
• Requests without a matching template are grouped into the handler none.

If one of these presets does not suit your needs you can do one of multiple things:

• Pick one of the already existing closures from metrics and pass it to the instrumentator instance. See here how to do that.
• Create your own instrumentation function that you can pass to an instrumentator instance. See here to learn how more.
• Don't use this package at all and just use the source code as inspiration on how to instrument your FastAPI.

Table of Contents

• Disclaimer
• Features
• Advanced Usage
  • Creating the Instrumentator
  • Adding metrics
  • Creating new metrics
  • Perform instrumentation
  • Specify namespace and subsystem
  • Exposing endpoint
• Contributing
• Licensing

Disclaimer

Not made for generic Prometheus instrumentation in Python. Use the Prometheus client library for that. This packages uses it as well.

All the generic middleware and instrumentation code comes with a cost in performance that can become noticeable.

Features

Beyond the fast track, this instrumentator is highly configurable and it is very easy to customize and adapt to your specific use case. Here is a list of some of these options you may opt-in to:

• Regex patterns to ignore certain routes.
• Completely ignore untemplated routes.
• Control instrumentation and exposition with an env var.
• Rounding of latencies to a certain decimal number.
• Renaming of labels and the metric.
• Metrics endpoint can compress data with gzip.
• Opt-in metric to monitor the number of requests in progress.

It also features a modular approach to metrics that should instrument all FastAPI endpoints. You can either choose from a set of already existing metrics or create your own. And every metric function by itself can be configured as well.

Advanced Usage

This chapter contains an example on the advanced usage of the Prometheus FastAPI Instrumentator to showcase most of it's features.

Creating the Instrumentator

We start by creating an instance of the Instrumentator. Notice the additional metrics import. This will come in handy later.

from prometheus_fastapi_instrumentator import Instrumentator, metrics

instrumentator = Instrumentator(
    should_group_status_codes=False,
    should_ignore_untemplated=True,
    should_respect_env_var=True,
    should_instrument_requests_inprogress=True,
    excluded_handlers=[".*admin.*", "/metrics"],
    env_var_name="ENABLE_METRICS",
    inprogress_name="inprogress",
    inprogress_labels=True,
)

Unlike in the fast track example, now the instrumentation and exposition will only take place if the environment variable ENABLE_METRICS is true at run-time. This can be helpful in larger deployments with multiple services depending on the same base FastAPI.

Adding metrics

Let's say we also want to instrument the size of requests and responses. For this we use the add() method. This method does nothing more than taking a function and adding it to a list. Then during run-time every time FastAPI handles a request all functions in this list will be called while giving them a single argument that stores useful information like the request and response objects. If no add() at all is used, the default metric gets added in the background. This is what happens in the fast track example.

All instrumentation functions are stored as closures in the metrics module.

Closures come in handy here because it allows us to configure the functions within.

instrumentator.add(metrics.latency(buckets=(1, 2, 3,)))

This simply adds the metric you also get in the fast track example with a modified buckets argument. But we would also like to record the size of all requests and responses.

instrumentator.add(
    metrics.request_size(
        should_include_handler=True,
        should_include_method=False,
        should_include_status=True,
        metric_namespace="a",
        metric_subsystem="b",
    )
).add(
    metrics.response_size(
        should_include_handler=True,
        should_include_method=False,
        should_include_status=True,
        metric_namespace="namespace",
        metric_subsystem="subsystem",
    )
)

You can add as many metrics you like to the instrumentator.

Creating new metrics

As already mentioned, it is possible to create custom functions to pass on to add(). This is also how the default metrics are implemented.

The basic idea is that the instrumentator creates an info object that contains everything necessary for instrumentation based on the configuration of the instrumentator. This includes the raw request and response objects but also the modified handler, grouped status code and duration. Next, all registered instrumentation functions are called. They get info as their single argument.

Let's say we want to count the number of times a certain language has been requested.

from typing import Callable
from prometheus_fastapi_instrumentator.metrics import Info
from prometheus_client import Counter

def http_requested_languages_total() -> Callable[[Info], None]:
    METRIC = Counter(
        "http_requested_languages_total",
        "Number of times a certain language has been requested.",
        labelnames=("langs",)
    )

    def instrumentation(info: Info) -> None:
        langs = set()
        lang_str = info.request.headers["Accept-Language"]
        for element in lang_str.split(","):
            element = element.split(";")[0].strip().lower()
            langs.add(element)
        for language in langs:
            METRIC.labels(language).inc()

    return instrumentation

The function http_requested_languages_total is used for persistent elements that are stored between all instrumentation executions (for example the metric instance itself). Next comes the closure. This function must adhere to the shown interface. It will always get an Info object that contains the request, response and a few other modified informations. For example the (grouped) status code or the handler. Finally, the closure is returned.

Important: The response object inside info can either be the response object or None. In addition, errors thrown in the handler are not caught by the instrumentator. I recommend to check the documentation and/or the source code before creating your own metrics.

To use it, we hand over the closure to the instrumentator object.

instrumentator.add(http_requested_languages_total())

Perform instrumentation

Up to this point, the FastAPI has not been touched at all. Everything has been stored in the instrumentator only. To actually register the instrumentation with FastAPI, the instrument() method has to be called.

instrumentator.instrument(app)

Notice that this will do nothing if should_respect_env_var has been set during construction of the instrumentator object and the respective env var is not found.

Specify namespace and subsystem

You can specify the namespace and subsystem of the metrics by passing them in the instrument method.

from prometheus_fastapi_instrumentator import Instrumentator

@app.on_event("startup")
async def startup():
    Instrumentator().instrument(app, metric_namespace='myproject', metric_subsystem='myservice').expose(app)

Then your metrics will contain the namespace and subsystem in the metric name.

# TYPE myproject_myservice_http_request_duration_highr_seconds histogram
myproject_myservice_http_request_duration_highr_seconds_bucket{le="0.01"} 0.0

Exposing endpoint

To expose an endpoint for the metrics either follow Prometheus Python Client and add the endpoint manually to the FastAPI or serve it on a separate server. You can also use the included expose method. It will add an endpoint to the given FastAPI. With should_gzip you can instruct the endpoint to compress the data as long as the client accepts gzip encoding. Prometheus for example does by default. Beware that network bandwith is often cheaper than CPU cycles.

instrumentator.expose(app, include_in_schema=False, should_gzip=True)

Notice that this will to nothing if should_respect_env_var has been set during construction of the instrumentator object and the respective env var is not found.

Contributing

Please refer to CONTRIBUTING.md.

Consult DEVELOPMENT.md for guidance regarding development.

Read RELEASE.md for details about the release process.

Licensing

The default license for this project is the ISC License. A permissive license functionally equivalent to the BSD 2-Clause and MIT licenses, removing some language that is no longer necessary. See LICENSE for the license text.

The BSD 3-Clause License is used as the license for the routing module. This is due to it containing code from elastic/apm-agent-python. BSD 3-Clause is a permissive license similar to the BSD 2-Clause License, but with a 3rd clause that prohibits others from using the name of the copyright holder or its contributors to promote derived products without written consent. The license text is included in the module itself.

prometheus-fastapi-instrumentator's Issues

Body not present on instrumentation info.response object

Hi!

When calling the instrumentation function inside my custom metric, the response attribute of the info parameter does not have body.

def whatever() -> Callable[[Info], None]:
    metric = Gauge("whatever",
                   "This metric tracks whatever api usage.")

    def instrumentation(info: Info) -> None:
        request = info.request
        response = info.response
        body = response.body

    return instrumentation

AttributeError: 'StreamingResponse' object has no attribute 'body'

Is there a way to circumvent this issue or maybe I'm doing sth wrong!

Thank you all in advance!

pip install warning: prometheus-fastapi-instrumentator 5.4.0 requires fastapi==0.38.1, but you'll have fastapi 0.61.1 which is incompatible.

Shouldn't it be >=0.38.1 in requirements in pyproject.toml
https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/pyproject.toml#L17

See warning at end below

$ pip install -U fastapi
Collecting fastapi
  Using cached fastapi-0.61.1-py3-none-any.whl (48 kB)
Collecting pydantic<2.0.0,>=1.0.0
  Using cached pydantic-1.6.1-cp38-cp38-macosx_10_9_x86_64.whl (2.3 MB)
Collecting starlette==0.13.6
  Using cached starlette-0.13.6-py3-none-any.whl (59 kB)
Installing collected packages: pydantic, starlette, fastapi
  Attempting uninstall: pydantic
    Found existing installation: pydantic 0.32.2
    Uninstalling pydantic-0.32.2:
      Successfully uninstalled pydantic-0.32.2
  Attempting uninstall: starlette
    Found existing installation: starlette 0.12.8
    Uninstalling starlette-0.12.8:
      Successfully uninstalled starlette-0.12.8
  Attempting uninstall: fastapi
    Found existing installation: fastapi 0.38.1
    Uninstalling fastapi-0.38.1:
      Successfully uninstalled fastapi-0.38.1
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

prometheus-fastapi-instrumentator 5.4.0 requires fastapi==0.38.1, but you'll have fastapi 0.61.1 which is incompatible.
Successfully installed fastapi-0.61.1 pydantic-1.6.1 starlette-0.13.6

Example for "manually" pushing a metric

I'd love to see an example of how to "manually" submit a metric, something like:

pseudocode:

@app.get('/super-route')
async def super-thing():
     business_result = await call_some_business_logic()
     metrics.push(business_result.count)

it's difficult to see how to "interact" with the metrics dynamically in code without going through the request/response object.

http_requests_total is only available as a default metric

Hello, I noticed that the default metrics contain the metric http_requests_total. As this metric is only defined inside the method default, it was necessary to create it as a custom metric:

def http_requests_total(metric_namespace='', metric_subsystem='') -> Callable[[Info], None]:
    total = Counter(
        name="http_requests_total",
        documentation="Total number of requests by method, status and handler.",
        labelnames=(
            "method",
            "status",
            "handler",
        ),
        namespace=metric_namespace,
        subsystem=metric_subsystem,
    )

    def instrumentation(info: Info) -> None:
        total.labels(info.method, info.modified_status, info.modified_handler).inc()

    return instrumentation

It would be great to have this metric available as a method like latency and response_size.

Thanks!

Strange drops in total requests

Hi,

I'm getting strange drops in the http_requests_total metric for the "/metrics" endpoint. I was expecting a monotonic increase as with each scrape, the "/metrics" counter should increase by one.

But it looks like that:

Any idea what I'm doing wrong?

Thanks and BR
Simon

Instrumentator Removes Causes from Exceptions

Affected Version: 5.7.1

Description:

When an exception is thrown somewhere down the line, the Prometheus Instrumentator removes all cause information from the Exception.

Is this a deliberate design choice or an oversight?

Cause

The middleware is implemented like this: (instrumentation.py line 168-172)

            try:
                response = await call_next(request)
                status = str(response.status_code)
            except Exception as e:
                raise e from None

The from None removes information about what caused the exception.

Respective python docs: https://docs.python.org/3.8/reference/simple_stmts.html#raise

Optional summary with one or two most common percentiles

Add option to enable instrumentation with summary per endpoint for one or two percentiles. Should be straight-forward to implement without introducing breaking changes.

Trying to make custom Gauge

Hi. I'm having trouble with creating a custom gauge to count a number of open sockets. No matter what I do the metric doesn't seem to return a result:

def active_connections_number() -> Callable[[Info], None]:
    metric = Gauge(
        name="active_connections_number",
        documentation="Number of open websockets on this instance,",
        labelnames=["ocpp_active_connections_number"],
    )

    def instrumentation(info: Info) -> None:
        metric.set(value=3)

    return instrumentation

Instrumentator().add(active_connections_number())
Instrumentator().instrument(app).expose(app)

Doe this look like it should work correctly ?
I have no use for the Info but without it the code doesn't build.
Fastapi v 0.73.0, Python 3.7.10
Thanks

Project still uses prometheus_multiproc_dir environment variable in lower-case

To remain consistent with the Prometheus Python client it would be great if the uppercase version of this environment variable is used, as this is likely what people (like myself) would be trying to use based on the Prometheus documentation.

As per https://github.com/prometheus/client_python/releases/tag/v0.10.0

[CHANGE] The prometheus_multiproc_dir environment variable is deprecated in favor of PROMETHEUS_MULTIPROC_DIR. #624 - prometheus/client_python#624

The lowercase version is still used in

prometheus-fastapi-instrumentator/prometheus_fastapi_instrumentator/instrumentation.py

Line 257 in 45d0ba1

if "prometheus_multiproc_dir" in os.environ:

Passthrough arbitrary args / kwargs from expose() to FastAPI decorator

This way the parameters don't have to cover every possible use case

Performance Issues - Instrumentation is Resulting in Increased Latency

We are trying to decrease the latency of our BERT model prediction service that is deployed using FastAPI. The predictions are called through the /predict endpoint. We looked into the tracing and found one of the bottlenecks is the prometheus-fastapi-instrumentator. About 1% of the requests do timeout because they exceed 10s.

We also discovered that some metrics are not getting reported on 4 requests/second. Some requests took 30-50 seconds, with the starlette/fastapi taking long times. So it seems that under high usage, the /metrics endpoint doesn't get enough resources, and hence all /metrics requests wait for some time and fail eventually. So having separate container for metrics could help. Or if possible to have metrics delayed/paused under high load. Any insight/guidance would be much appreciated.

Metrics behind authentication middleware

Hello !

I would like to expose the metrics behind an authentication middleware. From the current API state, it doesn't feel like it's possible. Do you have some ideas how it could be achieved using prometheus-fastapi-instrumentator ? I am of course opened to adding this in the library if you think this is an interesting feature !

Thanks !

`Instrumentator` silenty adds metrics by default.

I can't turn them off, even with:

instrumentator.add(lambda info: None)

In order to try to prevent:

        if len(self.instrumentations) == 0:
            self.instrumentations.append(metrics.default())

It feels like too much magic for a regular user. This goes against the principles of least surprise and composability.

Is there a way how to add metrics that are not dependent on request/response?

Hello everyone, as title mentions, I wonder if this is something that can be done, if so how?

My present understanding is that the default implementation of metrics works on per requests/response basis. At least those objects are available from the prometheus_fastapi_instrumentator.metrics.Info object. I use them and they work as expected. However, the API I work on triggers fastapi.BackgroundTask with some external computation process, that is executed asynchronously.

What I would like to have is a metrics on how long the BackgroundTask execution took as well as resources consumed. I can extract the values when the task is completed, but I have no idea how should I create metric and pass those values to the instrumentator, so that these values are consumed in the same fashion as the default metrices.

Generally speaking the API contains end point for passing parameters. Response is immediate with info that the job has started (or not). Next, there is another end point to retrieve status / download results. If the job completed successfully no new job is triggered when new request comes in with the same parameters.

Any help would be greatly appreciated. Thank you!

Unnecessary tight version constraint limits FastAPI versions

Due to this line in the pyproject.toml file:

fastapi = "^0.38.1"

FastAPI versions newer than 0.38 cannot be used with this (current version of FastAPI is 0.75.2). When explicitly requesting a higher version the version solving fails (using poetry):

$ poetry update
Updating dependencies
Resolving dependencies... (0.0s)

  SolverProblemError

  Because prometheus-fastapi-instrumentator (5.8.0) depends on fastapi (>=0.38.1,<0.39.0)
   and no versions of prometheus-fastapi-instrumentator match >5.8.0,<6.0.0, prometheus-fastapi-instrumentator (>=5.8.0,<6.0.0) requires fastapi (>=0.38.1,<0.39.0).
  So, because my-repo depends on both fastapi (^0.75.0) and prometheus-fastapi-instrumentator (^5.8.0), version solving failed.

One solution would be relaxing the requirements:

fastapi = "^0.38"

or

fastapi = ">=0.38.1, <1.0.0"

`/metrics/` and `/metrics` return different things with the default configuration

Adding FastAPI tags to metrics route

Is there a way to customize where the metrics route is tagged in the generated FastAPI docs? I'm using tags to group routes, but my instrumented routes ('/metrics') always ends up in "default".

Accept truthy value for env var

The check for the value of env_var_name is currently not documented but it requires the value "true". I suggest that we accept other truthy values like True or 1, as well.

Allow multiple instrumentators for multiple servers

What is the problem?

If I run integration tests with my fastapi server instrumented with metrics I have the following error:

ValueError: Duplicated timeseries in CollectorRegistry: {'http_requests', 'http_requests_created', 'http_requests_total'}

How to reproduce?

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator

def test_expose_twice():
    app = FastAPI()
    Instrumentator().instrument(app).expose(app, endpoint="/metrics_1")
    Instrumentator().instrument(app).expose(app, endpoint="/metrics_2")

Issue

The issue is that the Instrumentator always use prometheus_client.REGISTRY for its metrics. When we run our tests we construct multiple instances of our servers so the instrumentator tries to register the same metrics for all of them.

Possible solution

Accept registry as a constructor parameter to PrometheusFastApiInstrumentator class so users can specify their own registries.

RuntimeErrors are masked

Hey! Thanks for putting so much work into this project. My team and I recently tried it for our newest backend. Unfortunately we observed that raise RuntimeError(e) gets masked by the instrumentator.

Example stacktrace with masked error:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/applications.py", line 180, in __call__
    await super().__call__(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 129, in dispatch_middleware
    instrumentation(info)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/prometheus_fastapi_instrumentator/metrics.py", line 462, in instrumentation
    int(info.response.headers.get("Content-Length", 0))
AttributeError: 'NoneType' object has no attribute 'headers'

Example stacktrace with desired error:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/applications.py", line 180, in __call__
    await super().__call__(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/routing.py", line 196, in app
    raw_response = await run_endpoint_function(
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/routing.py", line 147, in run_endpoint_function
    return await dependant.call(**values)
  File "app/main.py", line 52, in runtimerror
    raise RuntimeError("example message")
RuntimeError: example message

Project Status: Maintained without new features

I'm happy to see that people find use in this project. In 2020 I created it to handle instrumentation of a bunch of microservices I was working on. Since then I (mostly) moved on to other things.

The project is still maintained and good at doing what it's supposed to do. At the same time, please don't expect exciting new features anytime soon. If you know of any good alternatives, feel free to point them out. I'll gladly include them in the README.

Grafana dashboard template for displaying the default metrics

Hello,

most people use Prometheus together with Grafana. It would be great if there was a dashboard template hosted at https://grafana.com/grafana/dashboards which shows at least the default metrics from the library (custom ones could be added individually as desired). These dashboards can easily be imported into a running Grafana instance, hence saving users the hassle to create a dashboard on their own.

Cheers

Port to pure ASGI Middleware

Hi, I am a Starlette maintainer. We are trying to get folks moved away from BaseHTTPMiddleware because it has several unfixable bugs and performance issues. One of the highest impact ways of doing this is going to be porting downstream packages (like this one) to pure ASGI middleware.

Taking a look at the package, it seems like it would be pretty straightforward to port the middleware itself to a pure ASGI middleware. The part that is going to be tricky is that the API sends Request/Response objects into user's functions. That part can't really be changed. What I'd proposed is creating a Request object with a placeholder receive that raises an error if called (i.e. you can't access the Request body) and maybe a Response that can't be run (i.e. Response.__call__ raises an exception).

To be clear I am volunteering my time to do this, but I will need review, some discussion about the above issues, etc.

Does this sound reasonable @trallnag?

incompatibility with fastapi-versioning

Whenever I use the instrumentator and I use the fastapi-versionin lib, all my handlers match just my version. I think it's because of the prefix format here:

app = VersionedFastAPI(
    app,
    version_format="{major}",
    prefix_format="/v{major}",
    default_version=(1, 0),
)

I'm not sure this is fixable but I did want to report it here

How to prefix metric names?

In something like Dogstatsd you can prefix your metrics using the namespace argument.

statsd = DogStatsd(host=statd_host, port=statd_port, namespace="your_apps_name")

this way in prometheus the metrics can be separated by app. Is something like this possible here?

Persistent Metric Data

Is there a way to store the counters on disk so the values persist after restarted the application? I think I've seen this done in other Prometheus client libraries where it saves the metrics to sqlite files? If this isn't possible, maybe this is a good candidate for a feature request?

How to add metrics

Hello! Thanks for the projet 🙂

I read the README and I'm left wondering how I can add metrics to the already defined ones. Is this even possible yet? For example, I'd like to add the average transfered data size per request/response. How would I add this into the instrumentator?

Only instrument / expose if env var is set

Enhancement: Execute logic in Instrumen() and expose() only if os envvar is set.

Usecase: Instrumenting a "base" FastAPI that is used by many different other apps (running as distinct instances) by default without actually performing the instrumentation.

Modular approach for instrumenting FastAPI endpoints with custom metrics

Currently this package instruments the endpoints of a given FastAPI with a single metric automatically. In the case I want to have additional metrics over all my endpoints I would have to either do it by hand for each endpoint or create another middleware I can add to the app. It would be nice to have a more way with the Prometheus FastAPI Instrumentator that requires less code than the two mentioned options.

Mentioned by @pawamoy in issue #2.

Init metrics for all handlers at startup

Only if possible. Would make it possible to count unused endpoints

BaseHTTPMiddleware vs BackgroundTasks

@trallnag I've just noticed that we use @app.middleware('http') here, I should have been able to catch this earlier... Anyway, that decorator is implemented on top of BaseHTTPMiddleware, which has a problem: encode/starlette#919

Solution: change the implementation to a pure ASGI app/middleware.

PS.: I can open a PR with it, jfyk.

endpoint not being exposed right

Hi, great work with this library!

I noticed a minor bug in the expose function endpoint always being '/metrics'.

New metric: number of requests in progress

Hello, do you have plans to add more metrics in the future? It would be great to have a metric that shows the number of requests in progress.

Thanks!

Awaiting the request’s body for labels

Hey, thanks a lot for the project :)

I wanted to start by adding a label to http_requests_total that would be coming from the request’s body.
For that, I followed the documentation to add a custom metric (that I renamed because http_requests_total is already present).
It works fine, but then I struggle to deal with the request.
In Info, it is a Starlette Request, and if I want the body I need to await it, which turns the instrumentation closure asynchronous and it breaks like so:

lib/python3.9/site-packages/prometheus_fastapi_instrumentator/instrumentation.py:196: RuntimeWarning: coroutine 'requests_total.<locals>.instrumentation' was never awaited

The documentation features an example that uses the request’s headers, which are accessible without waiting, but I didn’t find how to do it for the request itself.

Could you please advise on how to proceed?

Thanks!

Resolved handler metrics?

In our api, we've chosen to implement an api version as a pydantic validated url variable. Something like @router.get("/{api_version}/status") where the api_version is validated and rejected or accepted.

Prometheus-fastapi-instrumentator works like any other non-versioned route, and is measured fine, but, the api_version is never resolved into the actual version that is passed along. Out metrics end up like this:

http_requests_total{handler="/{api_version}/status",method="GET",status="2xx"} 32.0

While this works for general endpoint analysis, collecting time, or usage based on the api version would be better. So we could tell how frequently /1.0.0/status was hit vs /1.1.0/status, as a contrived example.

Is there any way to get the handler logged to resolved the api_version so our stats would end up like:

http_requests_total{handler="/1.0.0/status",method="GET",status="2xx"} 30.0
http_requests_total{handler="/1.1.0/status",method="GET",status="2xx"} 2.0

If not, and theoretically possible, would there be enough interest in a PR submitted to add this behavior, optionally?

Histogram not working

I am getting an error when trying to generate histogram metrics from my fastapi ML Prediction Service.

Full stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/usr/local/lib/python3.7/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 196, in dispatch_middleware
    instrumentation(info)
  File "./app/monitoring.py", line 95, in instrumentation
    METRIC.observe(float(prob))
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/metrics.py", line 570, in observe
    self._sum.inc(amount)
AttributeError: 'Histogram' object has no attribute '_sum'

My instrumentation code:

def http_classification_model_proba(
    metric_name: str = "http_classification_model_proba",
    metric_doc: str = "confidence scores outputted by MNB classification model",
    metric_namespace: str = "",
    metric_subsystem: str = "",
    buckets=(0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, float("inf"))
) -> Callable[[Info], None]:
    METRIC = Histogram(
        metric_name,
        metric_doc,
        buckets=buckets,
        namespace=metric_namespace,
        subsystem=metric_subsystem,
        labelnames=('classes',)
    )

    def instrumentation(info: Info) -> None:
        if info.modified_handler == "/predict":
            prob = info.response.headers.get("X-model-proba")
            prediction = info.response.headers.get("X-model-predict")
            if prob:
                METRIC.observe(float(prob))
                METRIC.labels(prediction).inc()

    return instrumentation

Print in middleware component

In version 5.8.2 the PrometheusInstrumentatorMiddleware prints to stdout on every request, this should not be in the release:

https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/prometheus_fastapi_instrumentator/middleware.py#L80

Workaround: Downgrade to 5.8.1

How to avoid missing metrics until first request is received?

many of the metrics, for example http_requests_total, are not showing up until the first request is received, this goes against prometheus best practices https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics

Duplicated Timeseries issue after 5.8.2 upgrade

Hey,
So my fastapi project automatically upgraded to the latest version of this project, and with 5.8.2 my server is failing to start with the following trace:

Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 437, in main
    run(app, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 463, in run
    server.run()
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/wg/vehicles/./api/main.py", line 50, in <module>
    async def setup_log_session(request: Request, call_next):
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 198, in decorator
    self.add_middleware(BaseHTTPMiddleware, dispatch=func)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 127, in add_middleware
    self.middleware_stack = self.build_middleware_stack()
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 91, in build_middleware_stack
    app = cls(app=app, **options)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 52, in __init__
    self.instrumentations = instrumentations or [metrics.default()]
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/metrics.py", line 544, in default
    TOTAL = Counter(
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/metrics.py", line 121, in __init__
    registry.register(self)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/registry.py", line 29, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'http_requests_created', 'http_requests_total', 'http_requests'}

Round to x decimals

Add more info to docs regarding multiproc mode

Make it more clear how the multiproc mode works and explicitly point out and link to the respective section in the prometheus client docs

Process Metrics unavailable to the MultiProcess CollectorRegistry

We are looking to get the process metrics such as process_resident_memory_bytes and process_cpu_seconds_total added to our FastAPI metrics endpoint. We are using the MultiProcessCollector which invokes CollectorRegistry and it looks like this doesn't allow for adding the ProcessCollector which generates those metrics. If for testing purposes I manually edit instrumentation.py and register PROCESSCOLLECTOR I can see the desired metrics in the output.

Is there some overriding reason that PROCESSCOLLECTOR is unavailable to the MultiProcessCollector? Is there some other way to achieve including the process metrics in the output? If not, we can investigate submitting a PR that allows optional additional collectors to be passed into the expose method.

Optional gzip expose

How to integrate with k8s HPA

Hi, I am relatively new to Prometheus. I've 2 questions

1. Without a prometheus server how is this package working? I installed the lib and added the
   Instrumentator().instrument(app).expose(app)
   line in my FastAPI application and when I hit the /metrics endpoint in my localhost it was displaying the metrics. But My question is without a server how did it work?

2. How can I integrate this with k8s HPA? I want to scale my pods if I get more than a certain no. of requests per second. Please help. TIA

Handler only with api version

I have problem that i see handler data only with version "/v1" not "/v1/ready" in prometheus.

Application:

from fastapi import FastAPI
from fastapi_versioning import VersionedFastAPI, version
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI(title='MyApp')

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.get("/ready")
@version(1)
async def ready():
    [my code]

app = VersionedFastAPI(app,
       version_format='{major}',
       prefix_format='/v{major}')

Instrumentator().instrument(app).expose(app)

Prometheus query:
http_requests_total{service="myapp"}

Prometheus data:

http_requests_total{container="myapp", endpoint="http", handler="/v1", instance="10.1.212.14:80", job="myapp", method="GET", namespace="default", pod="myapp-7c556d7547-pj87z", service="myapp", status="2xx"} | 10416

http_requests_total{container="myapp", endpoint="http", handler="/v1", instance="10.1.212.14:80", job="myapp", method="GET", namespace="default", pod="myapp-7c556d7547-pj87z", service="myapp", status="4xx"} | 2

Any idea?

How to export metrics on a different port

Is it possible to export metrics on a different port instead of using application port?

Unexpected print in Middleware

Hello)
I found the print in library code:

prometheus-fastapi-instrumentator/prometheus_fastapi_instrumentator/middleware.py

Line 80 in 673b922

print(is_excluded)

Can you pls remove it or allow some customization of this behavior?)

Instrumentator appears to be broken with the latest FastAPI

When I bootstrap, a simple application with FastAPI >= 0.70.0 and add the instrumentator according to the documentation, it'll throw the following error:

Traceback (most recent call last):
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/server.py", line 68, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/server.py", line 76, in serve
    config.load()
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/config.py", line 448, in load
    self.loaded_app = import_from_string(self.app)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/fohlen/PyCharmProjects/api-data/sanic-api/fast_api.py", line 101, in <module>
    Instrumentator().instrument(app).expose(app, tags=["metrics"])
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 127, in instrument
    self.instrumentations.append(metrics.default())
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_fastapi_instrumentator/metrics.py", line 563, in default
    TOTAL = Counter(
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_client/metrics.py", line 136, in __init__
    registry.register(self)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_client/registry.py", line 29, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'http_requests_total', 'http_requests', 'http_requests_created'}

I suspect this is because the async library changed to anyio, but I'm not too deep into the code to submit a fix.

Duplicate mime type charset=utf-8 on Response Header

Hi,

thank you for the great library. I have ran into an issue when using the metrics endpoint. It seems like the response contains the charset mime-type twice, which makes it incompatible with some HTTP clients.

Response Headers

content-type: text/plain; version=0.0.4; charset=utf-8; charset=utf-8

I am using the docker image tiangolo/uvicorn-gunicorn-fastapi:python3.8 and

fastapi==0.61.2
prometheus-fastapi-instrumentator==5.5.0

Do you know how I can fix this issue?

Latency & Counter Metrics Not Detected By Prometheus

Hello all,

I've been using this package for an ML monitoring usecase and it's been super helpful! Unfortunately, I'm experiencing a lot of trouble getting prometheus to scrape my Counter metric and latency as well. I set up my fastapi repo using an application factory design pattern. It seems like Histogram and Summary are going through though.

Do you have any insight as to what the issue could be? Would really appreciate your guidance as I've been trying to figure this out for 3 days.

Here is my monitoring.py file: https://github.com/rileyhun/fastapi-ml-example/blob/main/app/core/monitoring.py

Reproducible example:

git clone https://github.com/rileyhun/fastapi-ml-example.git

docker build -t ${IMAGE_NAME}:${IMAGE_TAG} -f Dockerfile .
docker tag ${IMAGE_NAME}:${IMAGE_TAG} rhun/${IMAGE_NAME}:${IMAGE_TAG}
docker push rhun/${IMAGE_NAME}:${IMAGE_TAG}

minikube start --driver=docker --memory 4g --nodes 2
kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring

kubectl apply -f deployment/wine-model-local.yaml
kubectl port-forward svc/wine-model-service 8080:80

python api_call.py