cuda-networks / django-eb-sqs Goto Github PK
View Code? Open in Web Editor NEWSQS background worker for Django
License: MIT License
SQS background worker for Django
License: MIT License
In [2]: @task(queue_name="mysqs.fifo")
...: def echo(message):
...: print("SQS send! messag")
...:
In [3]: echo.delay(message='Hello World!',group_id="123",MessageGroupId="123")
---------------------------------------------------------------------------
ClientError Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/eb_sqs/aws/sqs_queue_client.py in add_message(self, queue_name, msg, delay)
77 else:
---> 78 raise ex
79 except QueueDoesNotExistException:
/usr/local/lib/python3.6/site-packages/eb_sqs/aws/sqs_queue_client.py in add_message(self, queue_name, msg, delay)
67 MessageBody=msg,
---> 68 DelaySeconds=delay
69 )
/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py in do_action(self, *args, **kwargs)
519 def do_action(self, *args, **kwargs):
--> 520 response = action(self, *args, **kwargs)
521
/usr/local/lib/python3.6/site-packages/boto3/resources/action.py in __call__(self, parent, *args, **kwargs)
82
---> 83 response = getattr(parent.meta.client, operation_name)(*args, **params)
84
/usr/local/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
356 # The "self" in this scope is referring to the BaseClient.
--> 357 return self._make_api_call(operation_name, kwargs)
358
/usr/local/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
675 error_class = self.exceptions.from_code(error_code)
--> 676 raise error_class(parsed_response, operation_name)
677 else:
ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.
During handling of the above exception, another exception occurred:
QueueClientException Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/eb_sqs/worker/worker.py in _enqueue_task(self, worker_task, delay, execute_inline, is_retry, count_retries)
103 else:
--> 104 self.queue_client.add_message(worker_task.queue, worker_task.serialize(), delay)
105 return None
/usr/local/lib/python3.6/site-packages/eb_sqs/aws/sqs_queue_client.py in add_message(self, queue_name, msg, delay)
81 except Exception as ex:
---> 82 raise QueueClientException(ex)
QueueClientException: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.
During handling of the above exception, another exception occurred:
QueueException Traceback (most recent call last)
<ipython-input-3-17fe155eac47> in <module>
----> 1 echo.delay(message='Hello World!',group_id="123",MessageGroupId="123")
/usr/local/lib/python3.6/site-packages/eb_sqs/decorators.py in wrapper(*args, **kwargs)
24
25 worker = WorkerFactory.default().create()
---> 26 return worker.delay(group_id, queue, func, args, kwargs, max_retries, pickle, delay, execute_inline)
27
28 return wrapper
/usr/local/lib/python3.6/site-packages/eb_sqs/worker/worker.py in delay(self, group_id, queue_name, func, args, kwargs, max_retries, use_pickle, delay, execute_inline)
73 # type: (unicode, unicode, Any, tuple, dict, int, bool, int, bool) -> Any
74 worker_task = WorkerTask(str(uuid.uuid4()), group_id, queue_name, func, args, kwargs, max_retries, 0, None, use_pickle)
---> 75 return self._enqueue_task(worker_task, delay, execute_inline, False, True)
76
77 def retry(self, worker_task, delay, execute_inline, count_retries):
/usr/local/lib/python3.6/site-packages/eb_sqs/worker/worker.py in _enqueue_task(self, worker_task, delay, execute_inline, is_retry, count_retries)
114 ex)
115
--> 116 raise QueueException()
117
118 @classmethod
QueueException:
MessageGroupId
is a required parameter while enqueue to a queue
I am sporadically getting a key error when eb_sqs tries to initialize the SQS client (see sentry link below). Possibly related to this issue?
https://sentry.io/share/issue/e36b386c2ea4494aa1145b3557872ad8/
AWESOME work on this project, btw.
Python 2 is deprecated, and it would be better to fix all the typing and other relevant stuff to python 3 format.
Also no need to run python 2 flaky tests.
Hi, this project seems super useful!
I'm interested in using it in a production system and have a few minor improvement candidates.
I'd love to hear your feedback and we can have a quick discussion around these points before I jump into implementing anything. If you have other ideas that seem more pressing we can collect them in this thread as well.
Initial ideas:
Let me know what you think, thanks!
Project looks awesome from the readme, but I can't use a project without a License.
Can the current contributors agree on a License and license the repo.
I'd like to suggest the use of one of the MIT, BSD, or Apache 2.0 licenses. But its obviously up to the current contributors to decide.
Edit:
I'm not a lawyer, but I think, @patrick91 @Schweigi @itaybleier, you will all have to agree on the license.
Hey Contributors,
First of all I want to say this project has been a very useful component in a project I worked on recently.
I wanted to bring your attention to discuss the future plans or roadmaps for this project as I have a some ideas to discuss and see what you think of it and if it is something can be done I am happy to work on it and share a pull request to be reviewed ๐
boto3
configurations available to be set in django.conf.settings
(prefixed: AWS_
)some of the django based libraries already acknowledges the settings variables and let user define it in their. i.e
django-storages
I came across this error
CredentialRetrievalError
Error when retrieving credentials from iam-role: Credential refresh failed, response did not contain: access_key, secret_key, token, expiry_time
where set higher values for AWS_METADATA_SERVICE_NUM_ATTEMPTS
& AWS_METADATA_SERVICE_TIMEOUT
could have helped.
ampq
based brokers.For 2nd part, I want to connect with a core contributor who can help me understand the project in detail so I can start contributing to the improvements on documentation as well.
Looking forwarded to hearing from you guys.
After going through the source code, I didn't see anything meant to handle system signals (maybe I missed it). Is there a way to halt reading from the queue but continue processing running tasks? If not, would it be worth trying to handle a SIGTERM, for example, in the main worker loop?
Loving the project!
Hello,
I just spent awhile trying to figure out why the library was no longer working after updating it. I did not realize that elastic beanstalk was no longer supported until I looked through the diffs of the README and saw references to elastic beanstalk were removed. It would be nice if the README made it clear that support was removed.
There is no release on pypi, would really be handy to be able to install this via pip :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.