ydf0509 / funboost Goto Github PK

python3.9虚拟环境，pip安装依赖后运行demo报如下错误

比如 celery 需要在 gunicorn 中配置 celery -A app.tasks worker -c 3， funboost 需要配置什么吗？

消息队列选择sqlachemy+mysql时程序运行正常但是在数据库看不到数据表和对应的数据

python: 3.8.7
funboost: 16.3
mysql: 5.7.29

例如 __main__里面 add.consume() 它似乎是不阻塞的, 我在后面print,它执行完了也没有后续代码了.

一般的话整个程序就会退出了,然而你这个没有.请问咋实现的呢

https://www.waynerv.com/cookiecutter-pypackage/tutorial/
python三方库开发模板

比如我想单独enqueue一个task, 然后在某个时机下dequeue出来进行处理

使用python3.10版本运行测试案例报错

问题已解决

报错情况：

  File "C:\Users\90957\Documents\2022-1\fastapitest\test.py", line 2, in <module>
    from funboost import boost, BrokerEnum
  File "C:\software\Anaconda\envs\funboost\lib\site-packages\funboost\__init__.py", line 12, in <module>
    from funboost.consumers.base_consumer import (ExceptionForRequeue, ExceptionForRetry,
  File "C:\software\Anaconda\envs\funboost\lib\site-packages\funboost\consumers\base_consumer.py", line 25, in <module>
    from collections import Callable
ImportError: cannot import name 'Callable' from 'collections' (C:\software\Anaconda\envs\funboost\lib\collections\__init__.py)

原因：Deprecated since version 3.9: collections.abc.Callable Callable在python3.9已弃用
解决方案：使用低于3.9的python

出错的环境的配置完整命令：

conda create -n env_name python
conda activate env_name 
pip install funboost --upgrade

使用的代码

import time
from funboost import boost, BrokerEnum


@boost("task_queue_name1", qps=5, broker_kind=BrokerEnum.PERSISTQUEUE)  # 入参包括20种，运行控制方式非常多，想得到的控制都会有。
def task_fun(x, y):
    print(f'{x} + {y} = {x + y}')
    time.sleep(3)  # 框架会自动并发绕开这个阻塞，无论函数内部随机耗时多久都能自动调节并发达到每秒运行 5 次 这个 task_fun 函数的目的。


if __name__ == "__main__":
    for i in range(100):
        task_fun.push(i, y=i * 2)  # 发布者发布任务
    task_fun.consume()  # 消费者启动循环调度并发消费任务

大佬，我在django项目中使用时，会阻塞django进程（可能我用的不对？），
我觉得应该不会阻塞才对，不知怎么改进了。。

1. 定义一个任务

from funboost import boost, BrokerEnum

@boost('run_queue', broker_kind=BrokerEnum.LOCAL_PYTHON_QUEUE, is_using_rpc_mode=True, log_level=50)
def backup_task(device_info):
    backup = BackupNetConfTask(device_info)
    return "success"

2. 在视图函数中调用任务

from tasks import backup_task
    ... ...
    @action(methods=['post'], detail=False)
    def run_backup_task(self, request, *args, **kwargs):
        """
        api/v1/devices/run_backup_task/
        :param request: 请求对象
        :param args: 元组参数
        :param kwargs: 字典参数
        :return:
        """
        device_infos = get_device_infos() 
        for device_info in device_infos:
            backup_task.pub(dict(device_info=device_info))  <------------------ 程序阻塞在这里了
        return CustomResponse(status=status.HTTP_200_OK, msg="run backup task success")

3. 启动消费者

from tasks import backup_task
backup_task.consume()

验证了一下，准备投入使用试试，发现还是一大堆不兼容的问题，主要体现在包上。

1. 建议使用 poetry 来进行依赖管理，及时更新对新版本的支持。
2. 建议移除无关的包，或者把一些中间件做成可选安装的。

兼容性问题用起来确实有点头大。

大佬， 我创建了2个任务，第一个负责下载，第二个负责处理；但是负责处理的任务依赖于第一个，而处理任务会有大量的IO，因此我设置了下载2个线程处理，而处理任务只开启了1个。于是在提交任务的函数中设置了获取下载的运行状态，要求回调结果，但是弹出下面的错误，简化代码如下：

from tasks import download_task, compress_task

def request(url, tempfn):
    '''
    url 为下载地址
    tempfn 为下载的本地路径
    '''
    download_status = download_task.push(url)
    if download_status.result:
        compress_task.push(tempfn)

错误：

raceback (most recent call last):

  File "/root/anaconda3/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f3b3c808af0>
    └ <DummyProcess(Thread-8, started daemon 139891555944192)>
  File "/root/anaconda3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f3b3c808820>
    └ <DummyProcess(Thread-8, started daemon 139891555944192)>
  File "/root/anaconda3/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <DummyProcess(Thread-8, started daemon 139891555944192)>
    │    │        │    └ (<_queue.SimpleQueue object at 0x7f3b2179af40>, <_queue.SimpleQueue object at 0x7f3b2179aa40>, None, (), None, False)
    │    │        └ <DummyProcess(Thread-8, started daemon 139891555944192)>
    │    └ <function worker at 0x7f3b217c71f0>
    └ <DummyProcess(Thread-8, started daemon 139891555944192)>
  File "/root/anaconda3/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    │     │       └ {}
                    │     └ (('reanalysis-era5-pressure-levels', {'variable': 'geopotential', 'year': '2020', 'month': '4', 'format': 'netcdf', 'area': '...
                    └ <function only_request at 0x7f3b23864d30>
> File "/root/anaconda3/lib/python3.9/site-packages/era5/cli.py", line 111, in only_request
    if downloas_status.result: # 为了阻塞线程
       │               └ <property object at 0x7f3b24a51450>
       └ <funboost.publishers.base_publisher.AsyncResult object at 0x7f3b2078e2b0>
  File "/root/anaconda3/lib/python3.9/site-packages/funboost/publishers/base_publisher.py", line 68, in result
    return self.get()
           │    └ <function AsyncResult.get at 0x7f3b24a52280>
           └ <funboost.publishers.base_publisher.AsyncResult object at 0x7f3b2078e2b0>
  File "/root/anaconda3/lib/python3.9/site-packages/funboost/publishers/base_publisher.py", line 64, in get
    raise HasNotAsyncResult
          └ <class 'funboost.publishers.base_publisher.HasNotAsyncResult'>

有些模块压根用不着，默认也安装了。

安装报错，我截几个关键的位置，不知道如何解决
Building wheels for collected packages: gevent
Building wheel for gevent (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for gevent (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [289 lines of output]
.................
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for gevent
Failed to build gevent
ERROR: Could not build wheels for gevent, which is required to install pyproject.toml-based projects

11111

其实就是一个需求, 我们调用某saas服务, 最高100qps 超过就会封禁一直断时间, 而且此saas的接口的qps限制是共用的

也就是我们写的 一部分task 加起来的qps 不得超过100 但是我们又得让他尽量靠近100 这样可以最大化利用资源

文档我看了两遍
但是好像没有 按组 限制qps的

恳请解答 感谢作者开源,优秀项目

根据指引 把实际项目附加到funboost项目下，方便调试
实际生产项目中使用了阿里云 SLS日志服务。

工程项目使用了dictConfig和get_logger(name)
funbootst使用了推荐的PYTHONPATH的方式导入工程
根据funboost的规划，如我我需要继续使用 阿里云 sls服务，我似乎应该导入ng_log.get_logger 使用logger.addHandler()
logger.addFilter()去实现，是这样的么？

hi
想请问一下funboost如何结合tornado实现异步非阻塞啊？ 我想使用tornado通过funboost向队列中发消息 然后异步的获取执行结果 然后结束响应。
thks

想请教linux下如何部署消费函数

当成一个service写一个脚本后台运行么？

lib/python3.9/site-packages/funboost/queues/sqla_queue.py", line 66, in SessionContext
def init(self, session: sqlalchemy.orm.session.Session):
NameError: name 'sqlalchemy' is not defined

我在运行例子的时候报错了

下载完之后100多m，能不能精简精简，把不必要的库都删掉，只保留必须的库，只是建议啊，希望国产这库越来越好！

我是使用ahttp模拟的请求，有几个时间段之前是没有消费任务

配置的是tcp，没有用redis
queue_name='127.0.0.1:5689', broker_kind=BrokerEnum.TCP
但是还是消费者那里报redis的错误
redis2.exceptions.ConnectionError: Error 10061 connecting to 127.0.0.1:6379. 由于目标计算机积极拒绝，无法连接。.

import time
from funboost import boost, BrokerEnum,get_consumer,run_consumer_with_multi_process

def f(a, b):
    return a+b

consumer = get_consumer('queue_test_f01', consuming_function=f,qps=0.2, broker_kind=2)

if __name__ == "__main__":
    # 需要手动指定consuming_function入参的值。
    for i in range(10, 2000):
        consumer.publisher_of_same_queue.publish(dict(a=i, b=i * 2))
        
    consumer.start_consuming_message()
    run_consumer_with_multi_process(consumer,8)
    

运行后，报错：

Traceback (most recent call last):
  File "main.py", line 16, in <module>
    run_consumer_with_multi_process(consumer,8)
  File "/home/project/test_mul_thread/venv/lib/python3.8/site-packages/funboost/helpers.py", line 45, in run_consumer_with_multi_process
    if not getattr(task_fun, 'is_decorated_as_consume_function'):
AttributeError: 'RedisConsumer' object has no attribute 'is_decorated_as_consume_function'

但不影响程序执行，程序会正常继续运行，想知道这个错误是否可以规避？

日志https://function-scheduling-distributed-framework.readthedocs.io/zh_CN/latest/articles/c2.html#id1

我想知道这是怎么导致的，环境变量中是导入过 export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

python 3.10.3 运行会出现这个哦
Traceback (most recent call last):
File "", line 1027, in _find_and_load
File "", line 1006, in find_and_load_unlocked
File "", line 688, in load_unlocked
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\python\python3.10\lib\site-packages\funboost_init.py", line 11, in
from funboost.assist.user_custom_broker_register import register_custom_broker
File "D:\python\python3.10\lib\site-packages\funboost\assist\user_custom_broker_register.py", line 2, in
from funboost.publishers.base_publisher import AbstractPublisher
File "D:\python\python3.10\lib\site-packages\funboost\publishers\base_publisher.py", line 22, in
from funboost.concurrent_pool import CustomThreadPoolExecutor
File "D:\python\python3.10\lib\site-packages\funboost\concurrent_pool_init.py", line 14, in
from .custom_evenlet_pool_executor import CustomEventletPoolExecutor
File "D:\python\python3.10\lib\site-packages\funboost\concurrent_pool\custom_evenlet_pool_executor.py", line 7, in
from eventlet import greenpool, monkey_patch, patcher, Timeout
File "D:\python\python3.10\lib\site-packages\eventlet_init.py", line 17, in
from eventlet import convenience
File "D:\python\python3.10\lib\site-packages\eventlet\convenience.py", line 7, in
from eventlet.green import socket
File "D:\python\python3.10\lib\site-packages\eventlet\green\socket.py", line 4, in
import('eventlet.green.socket_nodns')
File "D:\python\python3.10\lib\site-packages\eventlet\green_socket_nodns.py", line 11, in
from eventlet import greenio
File "D:\python\python3.10\lib\site-packages\eventlet\greenio_init.py", line 3, in
from eventlet.greenio.base import * # noqa
File "D:\python\python3.10\lib\site-packages\eventlet\greenio\base.py", line 32, in
socket_timeout = eventlet.timeout.wrap_is_timeout(socket.timeout)
File "D:\python\python3.10\lib\site-packages\eventlet\timeout.py", line 166, in wrap_is_timeout
base.is_timeout = property(lambda _: True)
TypeError: cannot set 'is_timeout' attribute of immutable type 'TimeoutError'

在 consumers/base_consumer.py 文件的 schedulal_task_with_no_block() 中，第 1097 行代码：
elif self._concurrent_mode == ConcurrentModeEnum.GEVENT:
是不是应该改为：
elif self._concurrent_mode == ConcurrentModeEnum.EVENTLET:

非常仰慕，非常好用
一个小愿望：setup中是否考虑使用extras_require
默认还可以安装所有组件，可以按需选装组件。
非常感谢作者的贡献

把这个当做异步接口，想知道怎么才能获取任务的状态？

大佬能否出个与django结合使用的最佳实践呢

@task_deco(queue_name='test', broker_kind=BrokerEnum.REDIS, is_send_consumer_hearbeat_to_redis=True)
def test(a, b):
return a + b

test.consume()
async_result = test.push(1, 0)
print(async_result.status_and_result)

大佬好，我将funboost集成到自己写的一个包里，位置在/root/rdown。这个包是我写的命令行，需要Python setup.py install后安装到/root/anaconda3/lib下进行命令行使用，这时喔该如何启动消费者呢？是在/root/rdown下还是/root/ansconda3/lib/site-packages/rdown下呢？

其实celery也是一样的问题，但是我各种调试不好celery的后台任务，每次命令行执行都OK，一旦按照官方文档systemctl后台部署就不对了

很有幸了解到这个项目。我想问一下，多任务依赖，怎么配备优先级，以及等待任务执行完成之后 再执行。
比如： 当前时间为 2点， 任务参数为1 点
那么我在2点执行的时候是1点数据，因为别的原因导致任务超时或者失败， 重新执行2点的任务。 但是可能任务参数 为3点了。 是否会保存当时执行的参数。 并且可以判断多任务依赖关系。 A->b/d/f->C

如 pywin32 在 mac 平台上就没有依赖，在使用 poetry 安装 funboost 时，遇到 pywin32 就无法安装成功

大佬能否出一个mqtt项目实例，文档中没有找到相关使用说明。

做了一个flask的服务，由于需要分别启动，所以不知道怎么打包部署。请问有方法解决这个问题吗？

粗略看了下，好像只有constant定义了，配置、工厂和其他地方都没有相关代码，是需要自己拓展吗？还是我没有留意到？

15:28:17 "/Users/zhangxiaodong/PycharmProjects/GameFiBridge/tasks/test.py:7" queue_test_666 的消费者
2022-02-07 15:28:20 - apscheduler.scheduler - "/Users/zhangxiaodong/PycharmProjects/GameFiBridge/venv/lib/python3.9/site-packages/apscheduler/schedulers/base.py:988" - _process_jobs - ERROR - Error submitting job "timing_publish_deco.._deco (trigger: interval[0:00:03], next run at: 2022-02-07 15:28:20 CST)" to executor "default"
Traceback (most recent call last):
File "/Users/zhangxiaodong/PycharmProjects/GameFiBridge/venv/lib/python3.9/site-packages/apscheduler/schedulers/base.py", line 979, in _process_jobs
executor.submit_job(job, run_times)
File "/Users/zhangxiaodong/PycharmProjects/GameFiBridge/venv/lib/python3.9/site-packages/apscheduler/executors/base.py", line 71, in submit_job
self._do_submit_job(job, run_times)
File "/Users/zhangxiaodong/PycharmProjects/GameFiBridge/venv/lib/python3.9/site-packages/apscheduler/executors/pool.py", line 28, in _do_submit_job
f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

@boost("register", broker_kind=BrokerEnum.REDIS_STREAM,
       concurrent_mode=ConcurrentModeEnum.ASYNC, log_level=20)
async def register(cookie_dict: dict, meta_dict: dict, mobile: str):
       pass

register.publish({'cookie_dict': cookie_dict, 'meta_dict': meta_dict, 'mobile': mobile},
                                 priority_control_config=PriorityConsumingControlConfig(countdown=40))

报错

Error submitting job "AsyncPoolExecutor.submit (trigger: date[2022-04-09 17:55:16 CST], next run at: 2022-04-09 17:55:16 CST)" to executor "default"
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\lib\site-packages\apscheduler\schedulers\base.py", line 979, in _process_jobs
    executor.submit_job(job, run_times)
  File "C:\ProgramData\Miniconda3\lib\site-packages\apscheduler\executors\base.py", line 71, in submit_job
    self._do_submit_job(job, run_times)
  File "C:\ProgramData\Miniconda3\lib\site-packages\apscheduler\executors\pool.py", line 28, in _do_submit_job
    f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
  File "C:\ProgramData\Miniconda3\lib\concurrent\futures\thread.py", line 161, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

刚才重新创建了环境，专门用于学习测试这个框架，装一个包，发现有很多依赖包：
Successfully installed AMQPStorm-2.7.1 Automat-20.2.0 Flask-2.1.1 Jinja2-3.1.1 MarkupSafe-2.1.1 Werkzeug-2.1.1 aiohttp-3.8.0 aiosignal-1.2.0 amqp-5.1.0 apscheduler-3.7.0 async-timeout-4.0.2 attrs-21.4.0 bcrypt-3.2.0 blinker-1.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 click-8.1.2 concurrent-log-handler-0.9.19 constantly-15.1.0 cryptography-36.0.2 decorator-4.4.0 deprecated-1.2.13 dnspython-1.16.0 dominate-2.6.0 elastic-transport-8.1.1 elasticsearch-8.1.2 eventlet-0.31.0 fabric2-2.6.0 flask-bootstrap-3.3.7.1 flask-login-0.6.0 flask-wtf-1.0.1 frozenlist-1.3.0 funboost-15.9 gevent-21.1.2 gnsq-1.0.1 greenlet-1.1.2 hyperlink-21.0.0 idna-3.3 importlib-metadata-4.11.3 incremental-21.3.0 invoke-1.7.0 itsdangerous-2.1.2 kafka-python-2.0.2 kombu-5.2.4 multidict-6.0.2 nats-python-0.8.0 nb-filelock-0.7 nb-log-7.5 packaging-21.3 paho-mqtt-1.6.1 pamqp-2.3.0 paramiko-2.10.3 pathlib2-2.3.7.post1 persist-queue-0.7.0 pika-1.2.0 pikav0-0.1.23 pikav1-1.0.13 portalocker-2.4.0 psutil-5.9.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pymongo-4.0.2 pynacl-1.5.0 pyparsing-3.0.7 pysnooper-1.1.0 python-json-logger-0.1.10 pytz-2022.1 pyzmq-22.3.0 rabbitpy-2.0.1 redis-4.2.1 redis2-2.10.6.3 redis3-3.5.2.3 requests-2.27.1 rocketmq-0.4.4 semantic-version-2.9.0 service-identity-21.1.0 setuptools-rust-1.2.0 six-1.16.0 sqlalchemy-1.3.10 sqlalchemy-utils-0.36.1 tomorrow3-1.1.0 tornado-6.1 twisted-22.2.0 typing-extensions-4.1.1 tzlocal-2.1 urllib3-1.26.9 vine-5.0.0 visitor-0.1.3 wrapt-1.14.0 wtforms-3.0.1 yarl-1.7.2 zipp-3.7.0 zmq-0.0.0 zope.event-4.5.0 zope.interface-5.4.0

建议：

1. 把不必要的依赖去掉；
2. 如果实在需要，是否可以按需引入；

报错运行截图

python version: 3.9
funboost version: 14.8

代码

from funboost import boost, BrokerEnum, ConcurrentModeEnum, FunctionResultStatusPersistanceConfig


@boost(
    "queue2",
    broker_kind=BrokerEnum.MONGOMQ,
    function_result_status_persistance_conf=FunctionResultStatusPersistanceConfig(True, True, 7*25*36000)
)
def print_hi(name):
    # Use a breakpoint in the code line below to debug your script.
    print(f'Hi, {name}')  # Press ⌘F8 to toggle the breakpoint.


# Press the green button in the gutter to run the script.
if __name__ == '__main__':
    for i in range(100):
        pass
    print_hi.push(f"Hello")

    print_hi.consume()
    # print_hi.consume()
    # print_hi.consume()

monbodb docker-compose

version: '3.1'

services:

  mongo:
    image: mongo
    restart: always
    ports:
      - 27017:27017
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example
    volumes:
      - ./data:/data/db

  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: example
      ME_CONFIG_MONGODB_URL: mongodb://root:example@mongo:27017/

mongo version:

➜  mongo docker exec cb5a5c4c8932 mongo --version
MongoDB shell version v5.0.5
Build Info: {
    "version": "5.0.5",
    "gitVersion": "d65fd89df3fc039b5c55933c0f71d647a54510ae",
    "openSSLVersion": "OpenSSL 1.1.1f  31 Mar 2020",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "ubuntu2004",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

• 提示

"C:\Python38\lib\site-packages\funboost\consumers\confirm_mixin.py:70" 这行代码没有获得redis锁 fsdf_lock__requeue_tasks_which_unconfirmed:class_sch_group_check

• 程序情况

1. 开了五个实例(进程)
2. 每个进程 qps=1, concurrent_num=1 (因为这个任务为CPU密集型
3. push很多任务,但同时运行的进程大约为 2-4
4. 后面竞争起来了,有效进程就为2了,然后参与锁竞争的进程好像还会满载cpu

非常好用的框架，之前一直在用celery，发现这个框架之后，我觉得无论从项目设计规划和上手难易度来说，这个框架都非常棒。

目前问题是，这个框架在使用python 3.10+的时候，存在一些不兼容，请问作者是否有计划加入pyhton 3.10的支持。

支持多种中间件 能否通过配置支持降级 [ kafka -> MQ - > redis -> 本地文件 ] 这种

taskA->taskB->taskC->taskD链式任务，taskC失败了，并重试了一定次数，修改taskC的bug后，是否可以单单恢复taskC，而且与taskD的链条不断。
看了celery，好像不支持，只是支持重试
类似下面的逻辑

task = Task(task_id=xxx)
print(task.state)
# FAILED
task.recover()

# or Task.recover_task(task_id)

我不打算用 celery，因为他没办法和mysql完美的结合，我打算测试一下你这个 产品

定时任务写法：
fsdf_background_scheduler.add_job(timing_publish_deco(xxx_spider), 'interval', id='xxx_add_queue', seconds=300)
fsdf_background_scheduler.add_job(timing_publish_deco(xxx_result), 'interval', id='xxx_result_queue', seconds=60)

日志：RedisConsumer--xxx_result_queue - "base_consumer.py:697" - INFO - 10 秒内执行了 2 次函数 [xxx_result ] ,函数平均运行耗时 0.009 秒

消费者：
xxx_spider.consume()
xxx_result.consume()

确实nb，但是吧，引入了好多项目用不上的包。flask、kafka、mogodb…