the middleware can't be installed for the actual FastAPI version about prometheus-fastapi-instrumentator

[trallnag]

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

[harochau]

can confirm, this works with the new Fastapi

instrumentator = Instrumentator().instrument(app)
@app.on_event("startup")
async def _startup():
    instrumentator.expose(app, endpoint='/metrics', tags=['metrics'])

[alfaro28]

Sure,, just a quick note, I'm using start_http_server because I run my prometheus server on a diferent port, but I asume you could save the Instrumentator variable and just call .expose on the startup_event function

from fastapi import FastAPI
from prometheus_client import start_http_server
from prometheus_fastapi_instrumentator import Instrumentator

async def startup_event():
    start_http_server(8080)
    
app = FastAPI(
    on_startup=[startup_event]
)

Instrumentator(
    excluded_handlers=["/metrics"],
).instrument(app)

[jonathanloske]

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

According to the maintainer:

Prometheus expects a fully initialized ASGI client, where uvicorn now delays execution in newer versions. Closing.

[alexted]

@trallnag thanks for the release.
with the new version I started to get another error when running unit tests:

tests/v1_1/advertisers/test_get_expenses.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1757: in get
    return await self.request(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1533: in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1620: in send
    response = await self._send_handling_auth(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1648: in _send_handling_auth
    response = await self._send_handling_redirects(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1685: in _send_handling_redirects
    response = await self._send_single_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1722: in _send_single_request
    response = await transport.handle_async_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_transports/asgi.py:162: in handle_async_request
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/fastapi/applications.py:271: in __call__
    await super().__call__(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py:118: in __call__
    await self.middleware_stack(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:184: in __call__
    raise exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:162: in __call__
    await self.app(scope, receive, _send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py:84: in __call__
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:108: in __call__
    response = await self.dispatch_func(request, call_next)
src/utils/middlewares/log_requests.py:19: in log_requests
    response_body = [chunk async for chunk in response.body_iterator]
src/utils/middlewares/log_requests.py:19: in <listcomp>
    response_body = [chunk async for chunk in response.body_iterator]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/middleware.py:181: in __call__
    instrumentation(info)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

info = <prometheus_fastapi_instrumentator.metrics.Info object at 0x7feb23667fd0>

    def instrumentation(info: Info) -> None:
>       TOTAL.labels(info.method, info.modified_status, info.modified_handler).inc()
E       NameError: free variable 'TOTAL' referenced before assignment in enclosing scope

../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/metrics.py:625: NameError

[AliSayyah]

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

Thanks. NameError is gone.

[Dencrash]

We have the same problem with both the fastapi versions 0.91.0 and 0.92.0 being effected. It seems like adding a middleware after startup is not valid anymore, though they also fixed the fact that middlewares are initialized multiple times. Is there an alternative way to integrate the instrumentator?

[duffn]

We're encountering the same issue. Here's where this was introduced in Starlette. encode/starlette@51c1de1#diff-3eb07c0eea72d98f1b844e07844c2a68b7c11c63956b67d5afdd9c7f79ab946fR135

[harochau]

I have the same problem.
Maybe we can use https://github.com/stephenhillier/starlette_exporter instead? Does anybody have experience with that library, is it a drop-in replacement?

[alexted]

Yes, I too am thinking of switching to another library.

[alfaro28]

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

[alexted]

This works until you run unit tests.

[alfaro28]

sorry I should have been more clear, what I'm currently doing is creating the instrumentator and "instrument" the app outside of the startup event, but actually "exposing" it on the startup event and seems to be working fine

[duffn]

sorry I should have been more clear, what I'm currently doing is creating the instrumentator and "instrument" the app outside of the startup event, but actually "exposing" it on the startup event and seems to be working fine

Can you share a bit of code here please?

[trallnag]

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

The README was updated to use startup around 6 months ago due to #80. It is probably best to update the README again with the next release. Just tried it (while testing other things) and for me it works without startup fine.

Here are relevant dependencies:

❯ pip list
Package                           Version Editable project location
--------------------------------- ------- ----------------------------
fastapi                           0.92.0
prometheus-client                 0.16.0
prometheus-fastapi-instrumentator 100.0.0
starlette                         0.25.0
uvicorn                           0.20.0

The code:

from fastapi import FastAPI

from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

@app.get("/app")
def read_main():
    return {"message": "Hello World from main app"}

Instrumentator().instrument(app).expose(app)

Reproducible examples would be great as I am not using the package actively anymore.

[alexted]

it breaks all my unit tests:

[trallnag]

@alexted does this also happen if you use https://github.com/stephenhillier/starlette_exporter? I think it is related to how the prometheus client library works. In the unit tests of this package here I call the following function at the beginning of every relevant unit test: https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/tests/helpers/utils.py#L4

Another solution seems to be to use a new registry for every unit test: rycus86/prometheus_flask_exporter#17

Current version of prometheus-fastapi-instrumentator does not support passing custom registry, but this feature has already been merged as part of clean up and I will release it today

[saschnet]

Is there anything pointing against the solution by @alfaro28 / @harochau?

For me, it is working with all circumstances (unit tests + the app itself) and is much simpler than adding a new registry for every unit test etc.

[trallnag]

@saschnet, I don't know 😆 I will update the README and switch to the example provided by @harochau

[alexted]

@alexted does this also happen if you use https://github.com/stephenhillier/starlette_exporter?

No, everything works fine when using this library. The application runs and the unit tests pass.

[AliSayyah]

For now, I'm excluding 'prometheus_fastapi_instrumentator' when running tests.
I'm using 'pytest' so incase anyone having the same problem:

instrumentor = None
    if "pytest" not in sys.modules:
        instrumentor = PrometheusFastApiInstrumentator(
            should_group_status_codes=False
        ).instrument(
            app,
        )

and:

    @app.on_event("startup")
    async def _startup() -> None: 
        ...
        if "pytest" not in sys.modules:
            setup_prometheus(app, instrumentor)
        ...

note: this would only work if you haven't imported 'pytest' outside your tests. there may be cleaner ways to check this.

[trallnag]

@alexted, probably related to #219

[alexted]

ty @trallnag