trallnag / prometheus-fastapi-instrumentator Goto Github PK

View Code? Open in Web Editor NEW

832.0 4.0 82.0 893 KB

Instrument your FastAPI with Prometheus metrics.

License: ISC License

Python 100.00%

prometheus fastapi metrics exporter instrumentation

prometheus-fastapi-instrumentator's Issues

`Instrumentator` silenty adds metrics by default.

I can't turn them off, even with:

instrumentator.add(lambda info: None)

In order to try to prevent:

        if len(self.instrumentations) == 0:
            self.instrumentations.append(metrics.default())

It feels like too much magic for a regular user. This goes against the principles of least surprise and composability.

Strange drops in total requests

Hi,

I'm getting strange drops in the http_requests_total metric for the "/metrics" endpoint. I was expecting a monotonic increase as with each scrape, the "/metrics" counter should increase by one.

But it looks like that:

Any idea what I'm doing wrong?

Thanks and BR
Simon

pip install warning: prometheus-fastapi-instrumentator 5.4.0 requires fastapi==0.38.1, but you'll have fastapi 0.61.1 which is incompatible.

Shouldn't it be >=0.38.1 in requirements in pyproject.toml
https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/pyproject.toml#L17

See warning at end below

$ pip install -U fastapi
Collecting fastapi
  Using cached fastapi-0.61.1-py3-none-any.whl (48 kB)
Collecting pydantic<2.0.0,>=1.0.0
  Using cached pydantic-1.6.1-cp38-cp38-macosx_10_9_x86_64.whl (2.3 MB)
Collecting starlette==0.13.6
  Using cached starlette-0.13.6-py3-none-any.whl (59 kB)
Installing collected packages: pydantic, starlette, fastapi
  Attempting uninstall: pydantic
    Found existing installation: pydantic 0.32.2
    Uninstalling pydantic-0.32.2:
      Successfully uninstalled pydantic-0.32.2
  Attempting uninstall: starlette
    Found existing installation: starlette 0.12.8
    Uninstalling starlette-0.12.8:
      Successfully uninstalled starlette-0.12.8
  Attempting uninstall: fastapi
    Found existing installation: fastapi 0.38.1
    Uninstalling fastapi-0.38.1:
      Successfully uninstalled fastapi-0.38.1
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

prometheus-fastapi-instrumentator 5.4.0 requires fastapi==0.38.1, but you'll have fastapi 0.61.1 which is incompatible.
Successfully installed fastapi-0.61.1 pydantic-1.6.1 starlette-0.13.6

Print in middleware component

In version 5.8.2 the PrometheusInstrumentatorMiddleware prints to stdout on every request, this should not be in the release:

https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/prometheus_fastapi_instrumentator/middleware.py#L80

Workaround: Downgrade to 5.8.1

Optional summary with one or two most common percentiles

Add option to enable instrumentation with summary per endpoint for one or two percentiles. Should be straight-forward to implement without introducing breaking changes.

Project Status: Maintained without new features

I'm happy to see that people find use in this project. In 2020 I created it to handle instrumentation of a bunch of microservices I was working on. Since then I (mostly) moved on to other things.

The project is still maintained and good at doing what it's supposed to do. At the same time, please don't expect exciting new features anytime soon. If you know of any good alternatives, feel free to point them out. I'll gladly include them in the README.

Metrics behind authentication middleware

Hello !

I would like to expose the metrics behind an authentication middleware. From the current API state, it doesn't feel like it's possible. Do you have some ideas how it could be achieved using prometheus-fastapi-instrumentator ? I am of course opened to adding this in the library if you think this is an interesting feature !

Thanks !

Example for "manually" pushing a metric

I'd love to see an example of how to "manually" submit a metric, something like:

pseudocode:

@app.get('/super-route')
async def super-thing():
     business_result = await call_some_business_logic()
     metrics.push(business_result.count)

it's difficult to see how to "interact" with the metrics dynamically in code without going through the request/response object.

Is there a way how to add metrics that are not dependent on request/response?

Hello everyone, as title mentions, I wonder if this is something that can be done, if so how?

My present understanding is that the default implementation of metrics works on per requests/response basis. At least those objects are available from the prometheus_fastapi_instrumentator.metrics.Info object. I use them and they work as expected. However, the API I work on triggers fastapi.BackgroundTask with some external computation process, that is executed asynchronously.

What I would like to have is a metrics on how long the BackgroundTask execution took as well as resources consumed. I can extract the values when the task is completed, but I have no idea how should I create metric and pass those values to the instrumentator, so that these values are consumed in the same fashion as the default metrices.

Generally speaking the API contains end point for passing parameters. Response is immediate with info that the job has started (or not). Next, there is another end point to retrieve status / download results. If the job completed successfully no new job is triggered when new request comes in with the same parameters.

Any help would be greatly appreciated. Thank you!

BaseHTTPMiddleware vs BackgroundTasks

@trallnag I've just noticed that we use @app.middleware('http') here, I should have been able to catch this earlier... Anyway, that decorator is implemented on top of BaseHTTPMiddleware, which has a problem: encode/starlette#919

Solution: change the implementation to a pure ASGI app/middleware.

PS.: I can open a PR with it, jfyk.

Handler only with api version

I have problem that i see handler data only with version "/v1" not "/v1/ready" in prometheus.

Application:

from fastapi import FastAPI
from fastapi_versioning import VersionedFastAPI, version
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI(title='MyApp')

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.get("/ready")
@version(1)
async def ready():
    [my code]

app = VersionedFastAPI(app,
       version_format='{major}',
       prefix_format='/v{major}')

Instrumentator().instrument(app).expose(app)

Prometheus query:
http_requests_total{service="myapp"}

Prometheus data:

http_requests_total{container="myapp", endpoint="http", handler="/v1", instance="10.1.212.14:80", job="myapp", method="GET", namespace="default", pod="myapp-7c556d7547-pj87z", service="myapp", status="2xx"} | 10416

http_requests_total{container="myapp", endpoint="http", handler="/v1", instance="10.1.212.14:80", job="myapp", method="GET", namespace="default", pod="myapp-7c556d7547-pj87z", service="myapp", status="4xx"} | 2

Any idea?

Allow multiple instrumentators for multiple servers

What is the problem?

If I run integration tests with my fastapi server instrumented with metrics I have the following error:

ValueError: Duplicated timeseries in CollectorRegistry: {'http_requests', 'http_requests_created', 'http_requests_total'}

How to reproduce?

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator

def test_expose_twice():
    app = FastAPI()
    Instrumentator().instrument(app).expose(app, endpoint="/metrics_1")
    Instrumentator().instrument(app).expose(app, endpoint="/metrics_2")

Issue

The issue is that the Instrumentator always use prometheus_client.REGISTRY for its metrics. When we run our tests we construct multiple instances of our servers so the instrumentator tries to register the same metrics for all of them.

Possible solution

Accept registry as a constructor parameter to PrometheusFastApiInstrumentator class so users can specify their own registries.

Round to x decimals

Grafana dashboard template for displaying the default metrics

Hello,

most people use Prometheus together with Grafana. It would be great if there was a dashboard template hosted at https://grafana.com/grafana/dashboards which shows at least the default metrics from the library (custom ones could be added individually as desired). These dashboards can easily be imported into a running Grafana instance, hence saving users the hassle to create a dashboard on their own.

Cheers

Latency & Counter Metrics Not Detected By Prometheus

Hello all,

I've been using this package for an ML monitoring usecase and it's been super helpful! Unfortunately, I'm experiencing a lot of trouble getting prometheus to scrape my Counter metric and latency as well. I set up my fastapi repo using an application factory design pattern. It seems like Histogram and Summary are going through though.

Do you have any insight as to what the issue could be? Would really appreciate your guidance as I've been trying to figure this out for 3 days.

Here is my monitoring.py file: https://github.com/rileyhun/fastapi-ml-example/blob/main/app/core/monitoring.py

Reproducible example:

git clone https://github.com/rileyhun/fastapi-ml-example.git

docker build -t ${IMAGE_NAME}:${IMAGE_TAG} -f Dockerfile .
docker tag ${IMAGE_NAME}:${IMAGE_TAG} rhun/${IMAGE_NAME}:${IMAGE_TAG}
docker push rhun/${IMAGE_NAME}:${IMAGE_TAG}

minikube start --driver=docker --memory 4g --nodes 2
kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring

kubectl apply -f deployment/wine-model-local.yaml
kubectl port-forward svc/wine-model-service 8080:80

python api_call.py

How to prefix metric names?

In something like Dogstatsd you can prefix your metrics using the namespace argument.

statsd = DogStatsd(host=statd_host, port=statd_port, namespace="your_apps_name")

this way in prometheus the metrics can be separated by app. Is something like this possible here?

Persistent Metric Data

Is there a way to store the counters on disk so the values persist after restarted the application? I think I've seen this done in other Prometheus client libraries where it saves the metrics to sqlite files? If this isn't possible, maybe this is a good candidate for a feature request?

How to add metrics

Hello! Thanks for the projet 🙂

I read the README and I'm left wondering how I can add metrics to the already defined ones. Is this even possible yet? For example, I'd like to add the average transfered data size per request/response. How would I add this into the instrumentator?

Duplicated Timeseries issue after 5.8.2 upgrade

Hey,
So my fastapi project automatically upgraded to the latest version of this project, and with 5.8.2 my server is failing to start with the following trace:

Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 437, in main
    run(app, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 463, in run
    server.run()
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 60, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 67, in serve
    config.load()
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.8/dist-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/wg/vehicles/./api/main.py", line 50, in <module>
    async def setup_log_session(request: Request, call_next):
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 198, in decorator
    self.add_middleware(BaseHTTPMiddleware, dispatch=func)
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 127, in add_middleware
    self.middleware_stack = self.build_middleware_stack()
  File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 91, in build_middleware_stack
    app = cls(app=app, **options)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/middleware.py", line 52, in __init__
    self.instrumentations = instrumentations or [metrics.default()]
  File "/usr/local/lib/python3.8/dist-packages/prometheus_fastapi_instrumentator/metrics.py", line 544, in default
    TOTAL = Counter(
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/metrics.py", line 121, in __init__
    registry.register(self)
  File "/usr/local/lib/python3.8/dist-packages/prometheus_client/registry.py", line 29, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'http_requests_created', 'http_requests_total', 'http_requests'}

`/metrics/` and `/metrics` return different things with the default configuration

Add more info to docs regarding multiproc mode

Make it more clear how the multiproc mode works and explicitly point out and link to the respective section in the prometheus client docs

Awaiting the request’s body for labels

Hey, thanks a lot for the project :)

I wanted to start by adding a label to http_requests_total that would be coming from the request’s body.
For that, I followed the documentation to add a custom metric (that I renamed because http_requests_total is already present).
It works fine, but then I struggle to deal with the request.
In Info, it is a Starlette Request, and if I want the body I need to await it, which turns the instrumentation closure asynchronous and it breaks like so:

lib/python3.9/site-packages/prometheus_fastapi_instrumentator/instrumentation.py:196: RuntimeWarning: coroutine 'requests_total.<locals>.instrumentation' was never awaited

The documentation features an example that uses the request’s headers, which are accessible without waiting, but I didn’t find how to do it for the request itself.

Could you please advise on how to proceed?

Thanks!

Unnecessary tight version constraint limits FastAPI versions

Due to this line in the pyproject.toml file:

fastapi = "^0.38.1"

FastAPI versions newer than 0.38 cannot be used with this (current version of FastAPI is 0.75.2). When explicitly requesting a higher version the version solving fails (using poetry):

$ poetry update
Updating dependencies
Resolving dependencies... (0.0s)

  SolverProblemError

  Because prometheus-fastapi-instrumentator (5.8.0) depends on fastapi (>=0.38.1,<0.39.0)
   and no versions of prometheus-fastapi-instrumentator match >5.8.0,<6.0.0, prometheus-fastapi-instrumentator (>=5.8.0,<6.0.0) requires fastapi (>=0.38.1,<0.39.0).
  So, because my-repo depends on both fastapi (^0.75.0) and prometheus-fastapi-instrumentator (^5.8.0), version solving failed.

One solution would be relaxing the requirements:

fastapi = "^0.38"

fastapi = ">=0.38.1, <1.0.0"

Trying to make custom Gauge

Hi. I'm having trouble with creating a custom gauge to count a number of open sockets. No matter what I do the metric doesn't seem to return a result:

def active_connections_number() -> Callable[[Info], None]:
    metric = Gauge(
        name="active_connections_number",
        documentation="Number of open websockets on this instance,",
        labelnames=["ocpp_active_connections_number"],
    )

    def instrumentation(info: Info) -> None:
        metric.set(value=3)

    return instrumentation

Instrumentator().add(active_connections_number())
Instrumentator().instrument(app).expose(app)

Doe this look like it should work correctly ?
I have no use for the Info but without it the code doesn't build.
Fastapi v 0.73.0, Python 3.7.10
Thanks

Body not present on instrumentation info.response object

Hi!

When calling the instrumentation function inside my custom metric, the response attribute of the info parameter does not have body.

def whatever() -> Callable[[Info], None]:
    metric = Gauge("whatever",
                   "This metric tracks whatever api usage.")

    def instrumentation(info: Info) -> None:
        request = info.request
        response = info.response
        body = response.body

    return instrumentation

AttributeError: 'StreamingResponse' object has no attribute 'body'

Is there a way to circumvent this issue or maybe I'm doing sth wrong!

Thank you all in advance!

Resolved handler metrics?

In our api, we've chosen to implement an api version as a pydantic validated url variable. Something like @router.get("/{api_version}/status") where the api_version is validated and rejected or accepted.

Prometheus-fastapi-instrumentator works like any other non-versioned route, and is measured fine, but, the api_version is never resolved into the actual version that is passed along. Out metrics end up like this:

http_requests_total{handler="/{api_version}/status",method="GET",status="2xx"} 32.0

While this works for general endpoint analysis, collecting time, or usage based on the api version would be better. So we could tell how frequently /1.0.0/status was hit vs /1.1.0/status, as a contrived example.

Is there any way to get the handler logged to resolved the api_version so our stats would end up like:

http_requests_total{handler="/1.0.0/status",method="GET",status="2xx"} 30.0
http_requests_total{handler="/1.1.0/status",method="GET",status="2xx"} 2.0

If not, and theoretically possible, would there be enough interest in a PR submitted to add this behavior, optionally?

Unexpected print in Middleware

Hello)
I found the print in library code:

prometheus-fastapi-instrumentator/prometheus_fastapi_instrumentator/middleware.py

Line 80 in 673b922

print(is_excluded)

Can you pls remove it or allow some customization of this behavior?)

How to integrate with k8s HPA

Hi, I am relatively new to Prometheus. I've 2 questions

Without a prometheus server how is this package working? I installed the lib and added the
Instrumentator().instrument(app).expose(app)
line in my FastAPI application and when I hit the /metrics endpoint in my localhost it was displaying the metrics. But My question is without a server how did it work?
How can I integrate this with k8s HPA? I want to scale my pods if I get more than a certain no. of requests per second. Please help. TIA

Process Metrics unavailable to the MultiProcess CollectorRegistry

We are looking to get the process metrics such as process_resident_memory_bytes and process_cpu_seconds_total added to our FastAPI metrics endpoint. We are using the MultiProcessCollector which invokes CollectorRegistry and it looks like this doesn't allow for adding the ProcessCollector which generates those metrics. If for testing purposes I manually edit instrumentation.py and register PROCESSCOLLECTOR I can see the desired metrics in the output.

Is there some overriding reason that PROCESSCOLLECTOR is unavailable to the MultiProcessCollector? Is there some other way to achieve including the process metrics in the output? If not, we can investigate submitting a PR that allows optional additional collectors to be passed into the expose method.

How to export metrics on a different port

Is it possible to export metrics on a different port instead of using application port?

Project still uses prometheus_multiproc_dir environment variable in lower-case

To remain consistent with the Prometheus Python client it would be great if the uppercase version of this environment variable is used, as this is likely what people (like myself) would be trying to use based on the Prometheus documentation.

As per https://github.com/prometheus/client_python/releases/tag/v0.10.0

[CHANGE] The prometheus_multiproc_dir environment variable is deprecated in favor of PROMETHEUS_MULTIPROC_DIR. #624 - prometheus/client_python#624

The lowercase version is still used in

prometheus-fastapi-instrumentator/prometheus_fastapi_instrumentator/instrumentation.py

Line 257 in 45d0ba1

if "prometheus_multiproc_dir" in os.environ:

Modular approach for instrumenting FastAPI endpoints with custom metrics

Currently this package instruments the endpoints of a given FastAPI with a single metric automatically. In the case I want to have additional metrics over all my endpoints I would have to either do it by hand for each endpoint or create another middleware I can add to the app. It would be nice to have a more way with the Prometheus FastAPI Instrumentator that requires less code than the two mentioned options.

Mentioned by @pawamoy in issue #2.

endpoint not being exposed right

Hi, great work with this library!

I noticed a minor bug in the expose function endpoint always being '/metrics'.

http_requests_total is only available as a default metric

Hello, I noticed that the default metrics contain the metric http_requests_total. As this metric is only defined inside the method default, it was necessary to create it as a custom metric:

def http_requests_total(metric_namespace='', metric_subsystem='') -> Callable[[Info], None]:
    total = Counter(
        name="http_requests_total",
        documentation="Total number of requests by method, status and handler.",
        labelnames=(
            "method",
            "status",
            "handler",
        ),
        namespace=metric_namespace,
        subsystem=metric_subsystem,
    )

    def instrumentation(info: Info) -> None:
        total.labels(info.method, info.modified_status, info.modified_handler).inc()

    return instrumentation

It would be great to have this metric available as a method like latency and response_size.

Thanks!

Only instrument / expose if env var is set

Enhancement: Execute logic in Instrumen() and expose() only if os envvar is set.

Usecase: Instrumenting a "base" FastAPI that is used by many different other apps (running as distinct instances) by default without actually performing the instrumentation.

Passthrough arbitrary args / kwargs from expose() to FastAPI decorator

This way the parameters don't have to cover every possible use case

Histogram not working

I am getting an error when trying to generate histogram metrics from my fastapi ML Prediction Service.

Full stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.7/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/usr/local/lib/python3.7/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 196, in dispatch_middleware
    instrumentation(info)
  File "./app/monitoring.py", line 95, in instrumentation
    METRIC.observe(float(prob))
  File "/usr/local/lib/python3.7/site-packages/prometheus_client/metrics.py", line 570, in observe
    self._sum.inc(amount)
AttributeError: 'Histogram' object has no attribute '_sum'

My instrumentation code:

def http_classification_model_proba(
    metric_name: str = "http_classification_model_proba",
    metric_doc: str = "confidence scores outputted by MNB classification model",
    metric_namespace: str = "",
    metric_subsystem: str = "",
    buckets=(0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, float("inf"))
) -> Callable[[Info], None]:
    METRIC = Histogram(
        metric_name,
        metric_doc,
        buckets=buckets,
        namespace=metric_namespace,
        subsystem=metric_subsystem,
        labelnames=('classes',)
    )

    def instrumentation(info: Info) -> None:
        if info.modified_handler == "/predict":
            prob = info.response.headers.get("X-model-proba")
            prediction = info.response.headers.get("X-model-predict")
            if prob:
                METRIC.observe(float(prob))
                METRIC.labels(prediction).inc()

    return instrumentation

New metric: number of requests in progress

Hello, do you have plans to add more metrics in the future? It would be great to have a metric that shows the number of requests in progress.

Thanks!

incompatibility with fastapi-versioning

Whenever I use the instrumentator and I use the fastapi-versionin lib, all my handlers match just my version. I think it's because of the prefix format here:

app = VersionedFastAPI(
    app,
    version_format="{major}",
    prefix_format="/v{major}",
    default_version=(1, 0),
)

I'm not sure this is fixable but I did want to report it here

Instrumentator appears to be broken with the latest FastAPI

When I bootstrap, a simple application with FastAPI >= 0.70.0 and add the instrumentator according to the documentation, it'll throw the following error:

Traceback (most recent call last):
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/server.py", line 68, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/server.py", line 76, in serve
    config.load()
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/config.py", line 448, in load
    self.loaded_app = import_from_string(self.app)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/Users/fohlen/.pyenv/versions/3.9.7/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/fohlen/PyCharmProjects/api-data/sanic-api/fast_api.py", line 101, in <module>
    Instrumentator().instrument(app).expose(app, tags=["metrics"])
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 127, in instrument
    self.instrumentations.append(metrics.default())
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_fastapi_instrumentator/metrics.py", line 563, in default
    TOTAL = Counter(
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_client/metrics.py", line 136, in __init__
    registry.register(self)
  File "/Users/fohlen/.pyenv/versions/sanic-api/lib/python3.9/site-packages/prometheus_client/registry.py", line 29, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'http_requests_total', 'http_requests', 'http_requests_created'}

I suspect this is because the async library changed to anyio, but I'm not too deep into the code to submit a fix.

Adding FastAPI tags to metrics route

Is there a way to customize where the metrics route is tagged in the generated FastAPI docs? I'm using tags to group routes, but my instrumented routes ('/metrics') always ends up in "default".

Instrumentator Removes Causes from Exceptions

Affected Version: 5.7.1

Description:

When an exception is thrown somewhere down the line, the Prometheus Instrumentator removes all cause information from the Exception.

Is this a deliberate design choice or an oversight?

Cause

The middleware is implemented like this: (instrumentation.py line 168-172)

            try:
                response = await call_next(request)
                status = str(response.status_code)
            except Exception as e:
                raise e from None

The from None removes information about what caused the exception.

Respective python docs: https://docs.python.org/3.8/reference/simple_stmts.html#raise

Accept truthy value for env var

The check for the value of env_var_name is currently not documented but it requires the value "true". I suggest that we accept other truthy values like True or 1, as well.

RuntimeErrors are masked

Hey! Thanks for putting so much work into this project. My team and I recently tried it for our newest backend. Unfortunately we observed that raise RuntimeError(e) gets masked by the instrumentator.

Example stacktrace with masked error:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/applications.py", line 180, in __call__
    await super().__call__(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 129, in dispatch_middleware
    instrumentation(info)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/prometheus_fastapi_instrumentator/metrics.py", line 462, in instrumentation
    int(info.response.headers.get("Content-Length", 0))
AttributeError: 'NoneType' object has no attribute 'headers'

Example stacktrace with desired error:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/applications.py", line 180, in __call__
    await super().__call__(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/routing.py", line 196, in app
    raw_response = await run_endpoint_function(
  File "[PATH TO VIRTUALENV]/lib/python3.8/site-packages/fastapi/routing.py", line 147, in run_endpoint_function
    return await dependant.call(**values)
  File "app/main.py", line 52, in runtimerror
    raise RuntimeError("example message")
RuntimeError: example message

Port to pure ASGI Middleware

Hi, I am a Starlette maintainer. We are trying to get folks moved away from BaseHTTPMiddleware because it has several unfixable bugs and performance issues. One of the highest impact ways of doing this is going to be porting downstream packages (like this one) to pure ASGI middleware.

Taking a look at the package, it seems like it would be pretty straightforward to port the middleware itself to a pure ASGI middleware. The part that is going to be tricky is that the API sends Request/Response objects into user's functions. That part can't really be changed. What I'd proposed is creating a Request object with a placeholder receive that raises an error if called (i.e. you can't access the Request body) and maybe a Response that can't be run (i.e. Response.__call__ raises an exception).

To be clear I am volunteering my time to do this, but I will need review, some discussion about the above issues, etc.

Does this sound reasonable @trallnag?

Performance Issues - Instrumentation is Resulting in Increased Latency

We are trying to decrease the latency of our BERT model prediction service that is deployed using FastAPI. The predictions are called through the /predict endpoint. We looked into the tracing and found one of the bottlenecks is the prometheus-fastapi-instrumentator. About 1% of the requests do timeout because they exceed 10s.

We also discovered that some metrics are not getting reported on 4 requests/second. Some requests took 30-50 seconds, with the starlette/fastapi taking long times. So it seems that under high usage, the /metrics endpoint doesn't get enough resources, and hence all /metrics requests wait for some time and fail eventually. So having separate container for metrics could help. Or if possible to have metrics delayed/paused under high load. Any insight/guidance would be much appreciated.

Response Headers

content-type: text/plain; version=0.0.4; charset=utf-8; charset=utf-8

I am using the docker image tiangolo/uvicorn-gunicorn-fastapi:python3.8 and

fastapi==0.61.2
prometheus-fastapi-instrumentator==5.5.0

Do you know how I can fix this issue?

trallnag / prometheus-fastapi-instrumentator Goto Github PK

prometheus-fastapi-instrumentator's Issues

What is the problem?

How to reproduce?

Issue

Possible solution

Description:

Cause

Recommend Projects

Recommend Topics

Recommend Org

Jobs