GithubHelp home page GithubHelp logo

django-eb-sqs's People

Contributors

alexeyts avatar holger-zeit avatar jacobevans avatar patrick91 avatar rohandev avatar schweigi avatar sherwinlu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-eb-sqs's Issues

MessageGroupId is a required parameter while enqueue to a queue


In [2]: @task(queue_name="mysqs.fifo") 
   ...: def echo(message): 
   ...:    print("SQS send! messag") 
   ...:                                                                                                                             

In [3]: echo.delay(message='Hello World!',group_id="123",MessageGroupId="123")                                                      
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/eb_sqs/aws/sqs_queue_client.py in add_message(self, queue_name, msg, delay)
     77                 else:
---> 78                     raise ex
     79         except QueueDoesNotExistException:

/usr/local/lib/python3.6/site-packages/eb_sqs/aws/sqs_queue_client.py in add_message(self, queue_name, msg, delay)
     67                     MessageBody=msg,
---> 68                     DelaySeconds=delay
     69                 )

/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py in do_action(self, *args, **kwargs)
    519             def do_action(self, *args, **kwargs):
--> 520                 response = action(self, *args, **kwargs)
    521 

/usr/local/lib/python3.6/site-packages/boto3/resources/action.py in __call__(self, parent, *args, **kwargs)
     82 
---> 83         response = getattr(parent.meta.client, operation_name)(*args, **params)
     84 

/usr/local/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
    358 

/usr/local/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    675             error_class = self.exceptions.from_code(error_code)
--> 676             raise error_class(parsed_response, operation_name)
    677         else:

ClientError: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

During handling of the above exception, another exception occurred:

QueueClientException                      Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/eb_sqs/worker/worker.py in _enqueue_task(self, worker_task, delay, execute_inline, is_retry, count_retries)
    103             else:
--> 104                 self.queue_client.add_message(worker_task.queue, worker_task.serialize(), delay)
    105                 return None

/usr/local/lib/python3.6/site-packages/eb_sqs/aws/sqs_queue_client.py in add_message(self, queue_name, msg, delay)
     81         except Exception as ex:
---> 82             raise QueueClientException(ex)

QueueClientException: An error occurred (MissingParameter) when calling the SendMessage operation: The request must contain the parameter MessageGroupId.

During handling of the above exception, another exception occurred:

QueueException                            Traceback (most recent call last)
<ipython-input-3-17fe155eac47> in <module>
----> 1 echo.delay(message='Hello World!',group_id="123",MessageGroupId="123")

/usr/local/lib/python3.6/site-packages/eb_sqs/decorators.py in wrapper(*args, **kwargs)
     24 
     25         worker = WorkerFactory.default().create()
---> 26         return worker.delay(group_id, queue, func, args, kwargs, max_retries, pickle, delay, execute_inline)
     27 
     28     return wrapper

/usr/local/lib/python3.6/site-packages/eb_sqs/worker/worker.py in delay(self, group_id, queue_name, func, args, kwargs, max_retries, use_pickle, delay, execute_inline)
     73         # type: (unicode, unicode, Any, tuple, dict, int, bool, int, bool) -> Any
     74         worker_task = WorkerTask(str(uuid.uuid4()), group_id, queue_name, func, args, kwargs, max_retries, 0, None, use_pickle)
---> 75         return self._enqueue_task(worker_task, delay, execute_inline, False, True)
     76 
     77     def retry(self, worker_task, delay, execute_inline, count_retries):

/usr/local/lib/python3.6/site-packages/eb_sqs/worker/worker.py in _enqueue_task(self, worker_task, delay, execute_inline, is_retry, count_retries)
    114                            ex)
    115 
--> 116             raise QueueException()
    117 
    118     @classmethod

QueueException:

MessageGroupId is a required parameter while enqueue to a queue

Drop python 2 support

Python 2 is deprecated, and it would be better to fix all the typing and other relevant stuff to python 3 format.
Also no need to run python 2 flaky tests.

Improvement candidates

Hi, this project seems super useful!

I'm interested in using it in a production system and have a few minor improvement candidates.
I'd love to hear your feedback and we can have a quick discussion around these points before I jump into implementing anything. If you have other ideas that seem more pressing we can collect them in this thread as well.

Initial ideas:

  1. Use CI to run tests on PRs and commits to master: useful as sanity check for commits and discussions about pull requests. Services like TravisCI are free for open-source projects like this
  2. Upload project to PyPI for easier installations via pip
  3. Allow explicit setting of credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) in django settings. If there not given, default to existing behaviour by boto discovering the credentials

Let me know what you think, thanks!

No license?

Project looks awesome from the readme, but I can't use a project without a License.
Can the current contributors agree on a License and license the repo.

I'd like to suggest the use of one of the MIT, BSD, or Apache 2.0 licenses. But its obviously up to the current contributors to decide.

Edit:
I'm not a lawyer, but I think, @patrick91 @Schweigi @itaybleier, you will all have to agree on the license.

Making boto3 related configs vars available to be set in Django settings

Hey Contributors,
First of all I want to say this project has been a very useful component in a project I worked on recently.

I wanted to bring your attention to discuss the future plans or roadmaps for this project as I have a some ideas to discuss and see what you think of it and if it is something can be done I am happy to work on it and share a pull request to be reviewed ๐Ÿ™Œ

  1. Making boto3 configurations available to be set in django.conf.settings (prefixed: AWS_)

some of the django based libraries already acknowledges the settings variables and let user define it in their. i.e django-storages

I came across this error

CredentialRetrievalError
Error when retrieving credentials from iam-role: Credential refresh failed, response did not contain: access_key, secret_key, token, expiry_time

where set higher values for AWS_METADATA_SERVICE_NUM_ATTEMPTS & AWS_METADATA_SERVICE_TIMEOUT could have helped.

  1. A little bit work on the documentation of this project. I see it as a highly efficient and scalable piece of technology with a complete and effective alternative to celery (as celery is not recommended to be used with SQS) and still I came across so many projects and people who want to use AWS SQS for it's reliability and scalability but celery doesn't work that well as it does for other ampq based brokers.

For 2nd part, I want to connect with a core contributor who can help me understand the project in detail so I can start contributing to the improvements on documentation as well.

Looking forwarded to hearing from you guys.

Handing SIGTERM in process_queues() command

After going through the source code, I didn't see anything meant to handle system signals (maybe I missed it). Is there a way to halt reading from the queue but continue processing running tasks? If not, would it be worth trying to handle a SIGTERM, for example, in the main worker loop?

Loving the project!

Make it clear that Elastic Beanstalk support was removed

Hello,

I just spent awhile trying to figure out why the library was no longer working after updating it. I did not realize that elastic beanstalk was no longer supported until I looked through the diffs of the README and saw references to elastic beanstalk were removed. It would be nice if the README made it clear that support was removed.

Release on pypi

There is no release on pypi, would really be handy to be able to install this via pip :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.