Propan is a powerful and easy-to-use Python framework for building event-driven applications that interact with any MQ Broker
Home Page: https://lancetnik.github.io/Propan/
License: MIT License
I'm a fullstack developer living and working in Russia. 🇷🇺
I have been building APIs and tools for Machine Learning and data systems, with different teams and organizations. 🌎
I created Propan built FastStream🚀 and currently spend a huge part of my time to work on it. 🤓
So, if my open source projects are useful for your product/company, please tweet with @diementros or product mention about - your feedback is very important for me.
Look for references at aiokafka documentation
Describe the bug
Body type in Kafka Broker
To Reproduce
import json
from propan import KafkaBroker, PropanApp
from pydantic import BaseModel
broker = KafkaBroker()
app = PropanApp(broker)
class Msg(BaseModel):
data: str
def des(val: bytes):
return json.loads(val.decode("utf-8"))
@broker.handle(
"test-topic", auto_offset_reset="latest", value_deserializer=des
)
async def hello(msg: Msg):
print(msg)
@app.after_startup
async def pub():
msg = Msg(data="hi")
await broker.publish(
msg.json(),
"test-topic",
)
...And steps to reproduce the behavior:
Expected behavior
I expect {"data": "hi"} in output
Environment
Running Propan 0.1.2.7 with CPython 3.10.10 on Linux
Additional context
It happens because PropanMessage expect only bytes in message body, but in this case it is dict.
Maybe need to catch and raise exceptions when handling messages, or add other types to PropanMessage body
Hello, thank you so mush for this library. Want to ask is it possibly to add support for ZeroMQ broker?
Needs to refactor RabbitMQ Test Broker message delivery logic at this code to accorde original RabbitMQ behavior.
All cases can be found at documentation
Add docstrings to all brokers public methods
Describe the bug
When I made a mistake specifying the argument type, I received an error.
To Reproduce
Add source code
from propan import KafkaBroker, PropanApp
broker = KafkaBroker()
app = PropanApp(broker)
class A:
...
@broker.handle("test-topic", auto_offset_reset="earliest")
async def hello(msg: str, a: A):
print(msg)Result:
2023-06-03 14:49:12,214 ERROR - Invalid args for response field! Hint: check that <class 'mode.A'> is a valid pydantic field type
2023-06-03 14:49:12,214 ERROR - Please, input module like [python_file:propan_app_name]
And steps to reproduce the behavior:
Expected behavior
More informative output, that tell something like "In the main.py file, on line 12, the argument 'a' has type A. It was expected to have type pydantic field and a default value of Depends or Context."
Environment
Running Propan 0.1.2.7 with CPython 3.10.10 on Linux
Need to replace logic there to merge __init__ and connect arguments to one kwargs. Connect args should has priority to replace default init args.
Expectred behavior:
from propan import RabbitBroker
broker = RabbitBroker(host="localhost", port=6379")
...
await broker.connect(host="127.0.0.1") # connects to 127.0.0.1:6379Description
Error while reconnecting to RabbitMQ. The reconnect happens fine thanks to aio_pika, but message consumption and processing stops.
To Reproduce
from propan import PropanApp
from propan import RabbitBroker
from propan.brokers.rabbit.schemas import RabbitQueue, RabbitExchange
broker = RabbitBroker("amqp://guest:guest@localhost:5672/")
app = PropanApp(broker)
@broker.handle(RabbitQueue(name="test", durable=True), RabbitExchange(name="kek", durable=True))
async def test_handler(body):
print(body)And steps to reproduce the behavior:
kek exchange. They will be published successfully, but the Propan app will not process them. It will only do this on app restartExpected behavior
After reconnecting to the broker, Propan should continue processing messages.
Environment
Running Propan 0.1.2.9 with CPython 3.11.2 on Linux
Additional context
I think the problem is caused by the consume method of aio_pika.abc.AbstractRobustQueue. If you consume messages in this way, after reconnecting aio_pika for some reason does not restore work.
We should prevent this obvious wrong behavior but don't block it totally. So, warning should be a better choice.
Needs to add this message and test it with pytest.warns
Current tests checks that PropanApp is placed correct and CLI can find it, but they doesn't know about broker startup correclty. This should be fixed.
Describe the bug
MyPy reports:
main.py:7: error: "event" of "RabbitRouter" does not return a value [func-returns-value]
@router.event("Foobar")
^~~~~~~~~~~~~~~~~~~
To Reproduce
Contents of main.py:
from fastapi import FastAPI
from propan.fastapi import RabbitRouter
router = RabbitRouter("amqp://user:password@localhost:5671")
app = FastAPI(lifespan=router.lifespan_context)
@router.event("Foobar")
async def foobar(message: str) -> None:
passMypy config:
[tool.mypy]
pretty = trueAnd steps to reproduce the behavior:
Run mypy main.py.
Expected behavior
There should be no type errors
Environment
Running Propan 0.1.3.6 with CPython 3.11.3 on Linux
Additional context
I looked at the code at https://github.com/Lancetnik/Propan/blob/main/propan/fastapi/core/router.py#L148 and cannot fathom where this type error comes from. To me it looks like everything has a return type. (Changing the handler to return a value also doesnt help).
Code shown:
import from propan.brokers.rabbit RabbitBroker, RabbitExchange
broker = RabbitBroker()
@broker.handler("test", exchange=RabbitExchange("test"))
asynchronous definition handler():
...
...
await broker.publish("Hi!", "test", exchange=RabbitExchange("test"))Few issues:
RabbitExchangeTo support plugins extension way we should support middlewares.
I think, they should be look like a parent class with different methods wrapping different stages in handler calling flow
With using polyfactory I suppose
An error occurred when upgrading from version 0.1.3.7 to 0.1.3.8.
ValidationError(model='my_handler', errors=[{'loc': ('body',), 'msg': 'field required', 'type': 'value_error.missing'}])
The error appears only with additional arguments in the handler function:
class Message(BaseModel):
data: str
async def my_handler(body: Message, logger: Logger, settings: Settings = Context()):
...And it does not occur if additional arguments are removed:
class Message(BaseModel):
data: str
async def my_handler(body: Message):
...The data type of the body does not affect the error, only the additional arguments of the function affect.
When rolling back to 0.1.3.7, the error stops reproducing.
I couldn't figure out what the problem was, since I'm not very immersed in the source code of the project. Thank you in advance for the answer.
Needs to move documentation endpoints to separated router and bind it to FastAPI app.
It is necessary to split BrokerRouter top-level Depends and documentation Depends.
First step to automatic AsyncAPI scheme generation - to build/find some examples, how this scheme should be looking for specific Propan Application.
Implement CLI command to generate valid asyncapi.yaml by PropanApp
Use this package to create one-page React Application, asking an AsyncAPI scheme as string from python backend at startup and represents it. This app will be builded to static files and used as an online PropanApp application frontend.
Hey there - love the direction of this project.
I'm curious what direction you are thinking in terms of how you might scale out event handlers. For instance, might it be possible to deploy multiple workers (processes or threads) for running message handlers.
Is the idea that each app would be the scope of a single microservice or would an app encompass multiple microservices (i.e. multiple handlers)?
Just trying to get a sense of the direction of the project. Thanks!
Exception generated by using broker.publish
@app.after_startup
async def setup(
logger: Logger,
broker: RabbitBroker,
):
r = await broker.publish(
FetchJob(url="https://example.com"),
queue='fetch',
)Results in this error:
(edgepath-py3.11) chrisgoddard@Chriss-MacBook-Pro ~/C/E/project (main) [1]> propan run project.services.serve:app
08:28:29.168 | DEBUG | 67154:MainThread | prefect.profiles - Using profile 'local'
2023-05-30 08:28:29,296 INFO - Propan app starting...
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 449, in format
return self._format(record)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 445, in _format
return self._fmt % values
~~~~~~~~~~^~~~~~~~
KeyError: 'exchange'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 1110, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 953, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 690, in format
s = self.formatMessage(record)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "project/.venv/lib/python3.11/site-packages/propan/log/formatter.py", line 79, in formatMessage
return super().formatMessage(record)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "project/.venv/lib/python3.11/site-packages/propan/log/formatter.py", line 67, in formatMessage
return super().formatMessage(record)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 659, in formatMessage
return self._style.format(record)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 451, in format
raise ValueError('Formatting field not found in record: %s' % e)
ValueError: Formatting field not found in record: 'exchange'
Call stack:
File "project/.venv/bin/propan", line 8, in <module>
sys.exit(cli())
File "project/.venv/lib/python3.11/site-packages/typer/main.py", line 311, in __call__
return get_command(self)(*args, **kwargs)
File "project/.venv/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "project/.venv/lib/python3.11/site-packages/typer/core.py", line 778, in main
return _main(
File "project/.venv/lib/python3.11/site-packages/typer/core.py", line 216, in _main
rv = self.invoke(ctx)
File "project/.venv/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "project/.venv/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "project/.venv/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "project/.venv/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper
return callback(**use_params) # type: ignore
File "project/.venv/lib/python3.11/site-packages/propan/cli/main.py", line 99, in run
_run(module=module, app=app, extra_options=extra, log_level=casted_log_level)
File "project/.venv/lib/python3.11/site-packages/propan/cli/main.py", line 133, in _run
asyncio.run(propan_app.run(log_level=app_level))
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
File "project/.venv/lib/python3.11/site-packages/propan/cli/app.py", line 81, in _start
await self._startup()
File "project/.venv/lib/python3.11/site-packages/propan/cli/app.py", line 103, in _startup
await func()
File "project/.venv/lib/python3.11/site-packages/fast_depends/usage.py", line 74, in async_typed_wrapper
await run_async(dependant.call, **solved_result)
File "project/.venv/lib/python3.11/site-packages/fast_depends/utils.py", line 36, in run_async
r = await func(*args, **kwargs)
File "project/.venv/lib/python3.11/site-packages/propan/utils/functions.py", line 20, in wrapper
r = await call_or_await(func, *args, **kwargs)
File "project/.venv/lib/python3.11/site-packages/fast_depends/utils.py", line 36, in run_async
r = await func(*args, **kwargs)
File "project/edgepath/services/serve.py", line 45, in setup
logger.info(r)
Message: <Basic.Ack object at 0x114fb7a20>
Arguments: ()```To begin with, I would like to say a great thanks to the developer, Propan is magnificent. Very long thought to simplify the work and bring in a convenient format, but never got it together.
I had some problems with the integration with FastAPI, probably due to incomplete documentation or other things.
FastAPI 0.96.0
Propan 0.1.2.12
from fastapi import FastAPI
from propan.fastapi import RabbitRouter
import uvicorn
app = FastAPI()
router = RabbitRouter("amqp://guest:guest@localhost:5672")
@router.get("/")
async def hello_http():
await router.broker.publish("Hello, Rabbit!", "test")
return "Hello, HTTP!"
app.include_router(router)
if __name__ == '__main__':
uvicorn.run(app)We get the following error:
line 12, in hello_http await router.broker.publish("Hello, Rabbit!", "test")
ValueError: RabbitBroker channel not started yet
This is because the "lifespan" is not cast to the router, but can only be put at the stage of creating the application. Unlike the deprecated "on_startup".
We fix this by adding a router lifespan at the stage of creating the application:
router = RabbitRouter("amqp://guest:guest@localhost:5672")
app = FastAPI(lifespan=router.lifespan_context)or send all events like example from tests:
async with router.lifespan_context(None):
await router.broker.publish("Hello, Rabbit!", "test")from propan.fastapi import RabbitRouter
from pydantic import BaseModel
from fastapi import FastAPI
import uvicorn
router = RabbitRouter("amqp://guest:guest@localhost:5672")
app = FastAPI(lifespan=router.lifespan_context)
class Message(BaseModel):
string: str
@router.get('/')
async def hello_http():
await router.broker.publish(Message(string='test'), "test")
return 'Hello, HTTP!'
@router.event('test')
async def hello(m: Message):
return True
app.include_router(router)
if __name__ == '__main__':
uvicorn.run(app)We get a pydantic validation error because we take a byte message and don't parse it.
Because the file "propan/fastapi/core/route.py" on line 35 has hardcode _raw=True
broker.handle(path, _raw=True, **handle_kwargs)(handler)
Since we don't convert bytes to dict, pydantic can't validate (which would be very convenient)
changing the _raw flag to False doesn't fix it, because the app method on line 75 is waiting for a "NativeMessage" object
async def app(message: NativeMessage) -> Any:
But the decode method only passes the dict contained in the .body
My crappy fix for the situation is _raw=False and add a check in the app method:
async def app(message: Union[NativeMessage, AnyDict]) -> Any:
if isinstance(message, dict):
body = message
headers = {}
else:
body: Union[AnyDict, bytes] = message.body
headers = message.headers
if first_arg is not None:
if not isinstance(body, dict): # pragma: no branch
body = {first_arg: body}
session = cls(body, headers)
else:
session = cls()
return await func(session)Hopefully we can find a good solution for easy integration with FastAPI
Describe the bug
Traceback (most recent call last):
File "/usr/local/bin/uvicorn", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 410, in main
run(
File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 578, in run
server.run()
File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
return asyncio.run(self.serve(sockets=sockets))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
config.load()
File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 473, in load
self.loaded_app = import_from_string(self.app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string
raise exc from None
File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
module = importlib.import_module(module_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/app/main.py", line 7, in
from propan import RabbitBroker
File "/usr/local/lib/python3.11/site-packages/propan/init.py", line 3, in
from propan.cli.app import * # noqa: F403
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/propan/cli/init.py", line 1, in
from propan.cli.main import cli
File "/usr/local/lib/python3.11/site-packages/propan/cli/main.py", line 10, in
from propan.cli.docs import docs_app
File "/usr/local/lib/python3.11/site-packages/propan/cli/docs/init.py", line 1, in
from propan.cli.docs.app import docs_app
File "/usr/local/lib/python3.11/site-packages/propan/cli/docs/app.py", line 12, in
from propan.cli.docs.serve import serve_docs
ModuleNotFoundError: No module named 'propan.cli.docs.serve'
To Reproduce
Almost the fastapi integration doc.
from propan import PropanApp
...And steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
Add propan -v command output to show your current project and system environment
Additional context
Add any other context about the problem here.
Describe the bug
In the latest version 0.1.3.0 there is an import for propan.cli.docs.serve, but the module is not included in the distribution.
Maybe that is because it is ignored here: https://github.com/Lancetnik/Propan/blob/main/propan/cli/docs/__init__.py#L3 ?
Curiously the gen.py exists:
$ ls […]/lib/python3.11/site-packages/propan/cli/docs/
app.py gen.py __init__.py
$ grep "version" […]/lib/python3.11/site-packages/propan/__about__.py
__version__ = "0.1.3.0"
Relevant traceback:
from propan.brokers.rabbit import RabbitBroker, RabbitQueue
/opt/pipx/home/venvs/poetry/lib/python3.11/site-packages/propan/__init__.py:3: in <module>
from propan.cli.app import * # noqa: F403
/opt/pipx/home/venvs/poetry/lib/python3.11/site-packages/propan/cli/__init__.py:1: in <module>
from propan.cli.main import cli
/opt/pipx/home/venvs/poetry/lib/python3.11/site-packages/propan/cli/main.py:10: in <module>
from propan.cli.docs import docs_app
/opt/pipx/home/venvs/poetry/lib/python3.11/site-packages/propan/cli/docs/__init__.py:1: in <module>
from propan.cli.docs.app import docs_app
/opt/pipx/home/venvs/poetry/lib/python3.11/site-packages/propan/cli/docs/app.py:12: in <module>
from propan.cli.docs.serve import serve_docs
E ModuleNotFoundError: No module named 'propan.cli.docs.serve'
Method should take List of SQSMessages and other SQS API arguments.
Method signature should be close to publish as possible.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.