earl / beanstalkc Goto Github PK

Keep things minimal; do not replicate the tutorial

hi, I use this lib in my project, in one process, I use a Connection to put message into a tube, after running several hours, I found that the process is blocked. I use strace -p to record the history, the last record are:

getpid()                                = 22953
sendto(17, "put 2147483648 0 120 353\r\n{\"user"..., 381, 0, NULL, 0) = 381
recvfrom(17,

obviously, the connection with beanstalkd blocked, and in beanstalkc.py, I found the code:

        self._socket.settimeout(None)
        self._socket_file = self._socket.makefile('rb')

the socket is block mode, I guess it is a possibility of blocking.

Collecting beanstalkc
  Using cached beanstalkc-0.4.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-ubj0ksew/beanstalkc/setup.py", line 5, in <module>
        from beanstalkc import __version__ as src_version
      File "/tmp/pip-build-ubj0ksew/beanstalkc/beanstalkc.py", line 42
        except socket.error, err:
                           ^
    SyntaxError: invalid syntax
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ubj0ksew/beanstalkc/

can u help me?

1. Start beanstalkd

2. Create the following script and run

   http://gist.github.com/239124

3. Kill the beanstalkd server

Result:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    job = beanstalk.reserve()
  File "/usr/lib/python2.6/site-packages/beanstalkc.py", line 109, in reserve
    ['DEADLINE_SOON', 'TIMED_OUT'])
  File "/usr/lib/python2.6/site-packages/beanstalkc.py", line 76, in interact_job
    jid, size = self.interact(command, expected_ok, expected_err)
  File "/usr/lib/python2.6/site-packages/beanstalkc.py", line 55, in interact
    status, results = self.read_response()
  File "/usr/lib/python2.6/site-packages/beanstalkc.py", line 65, in read_response
    return response[0], response[1:]
IndexError: list index out of range

Additional Information:

If I put a print above line 65 I can see that its getting an empty list and failing (obviously because response[1:] doesn't exist).

Not sure how you want to handle this, but raising an exception when the connection is lost is probably a good start.

[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]     "jobQueue_high"     :   bQueue.stats_tube('jobQueue_high'),
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]   File "/usr/local/lib/python2.7/dist-packages/beanstalkc.py", line 217, in stats_tube
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]     ['NOT_FOUND'])
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]   File "/usr/local/lib/python2.7/dist-packages/beanstalkc.py", line 118, in _interact_yaml
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]     size, = self._interact(command, expected_ok, expected_err)
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]   File "/usr/local/lib/python2.7/dist-packages/beanstalkc.py", line 86, in _interact
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]     SocketError.wrap(self._socket.sendall, command)
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]   File "/usr/local/lib/python2.7/dist-packages/beanstalkc.py", line 43, in wrap
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28]     raise SocketError(err)
[Tue Jan 13 18:53:44 2015] [error] [client 10.0.0.28] SocketError: [Errno 9] Bad file descriptor

Should return None if the tube doesn't exist

I did a quick modification for python 3.x compliance (See dnene@1acb0eb )

I haven't quite figured out how to run the existing test cases so created my own at test/test_beanstalkc.py

Could you describe what commands I need to run and from which directory to execute the test cases. That can help me work through any other issues.

The tests in test/test_beanstalkc.py are successful across both python 2.7 and 3.3

If you explicitly watch a single empty tube other than 'default', beanstalkc.Connection.peek_ready() will still return jobs from the 'default' tube.

This is an example, where, starting with a completely empty database, peek_ready() should not return a job, but does anyway:

import beanstalkc

beanstalk = beanstalkc.Connection(
    host='127.0.0.1',
    port=11301,
    )

assert beanstalk.using() == 'default'
beanstalk.put('test_job_default')

beanstalk.watch('xxx')
beanstalk.ignore('default')
assert beanstalk.watching() == ['xxx']

job = beanstalk.peek_ready()             # Should not get a job here:
assert job.body == 'test_job_default'

Versions:
beanstalkd = 1.9
beanstalkc = 0.4.0

Currently, beanstalkc for all practical purposes depends on PyYAML. Unless told otherwise, instantiating a Connection will try to import PyYAML; if that import fails, beanstalkc will log an error (using logging.error) but will continue working, by switching to a YAML-less operating mode.

I'm now convinced, that this failure mode is a mistake; it's the opposite of "fail-fast". In practice, the automatic fail-over to YAML-less operation has done more harm than good. Basically, it made (though typically brief) debugging necessary, instead of properly failing right away and making the error condition obvious.

Therefore, I plan to switch the default failure mode: in the future, when PyYAML is missing, beanstalkc won't handle the ImportError itself, but just pass it through. This is a more transparent fail-fast.

However, PyYAML won't become a hard dependency. You'll still be able to operate beanstalkc without PyYAML, explicitly switching to YAML-less operation by passing parse_yaml=False to the Connection constructor. Explicitly switching to YAML-less operation has been supported for a long time, so that's nothing new. It is already used by all real-world YAML-less deployments I know of.

So the main thing that'll change, is switching the default to a more sensible fail-fast behaviour.

Unless there is major opposition against this change, or someone suggests something better, the new default behaviour will be switched for the next feature release (beanstalkc 0.5.0).

I have implemented class called NetworkSafeConnection [1].

It handles two problems of regular Connection:

• It can reconnect, hiding socket exceptions from user of Connection, blocking execution until call happens successfully.. This is done in two ways (regulated by parameter block_until_success):
  1. Retrying infinitely until success it achieved
  2. Doing not more than MAX_RETRIES attempts, with delays of RETRY_DELAY_SECONDS
• It can restore state of used and watched tubes before connection was lost
• I have put down example (tests) in the NetworkSafeConnection.mkd [2]

The only problem is that if socket error happened while job was reserved and not deleted by say worker1, after server restarts it will be available for reservation for some other workers. But same time it can be deleted by worker1 when connection will be restored.

Please let me know how do you feel about such extension of functionality. Do you think others can benefit from it?

1. https://github.com/ysmolsky/beanstalkc
2. https://github.com/ysmolsky/beanstalkc/blob/master/NetworkSafeConnection.mkd

See beanstalkd commit d5d311.

The connection timeout and JOB_TO_BIG error handling is very useful and should be plenty battled tested by now.

Mind cutting a new PyPI release?

I'm using a snippet like this to put work on a tube in a beanstalk queue:

beanstalk = beanstalkc.Connection(host='localhost', port=11300)
beanstalk.use('built')
output = beanstalk.put('foo')
beanstalk.close() # this hangs

I can see that I'm producing to the tube by stating it. However, upon trying to reserve a job (without timeout) the reserve call hangs. Using peek_ready I can get a handle to the job.

>>> c = beanstalkc.Connection(host='localhost', port=11300)
>>> c.use('built') in c.tubes()
True
>>> c.stats_tube('built')
{'current-jobs-delayed': 0, 'pause': 0, 'name': 'built', 'cmd-pause-tube': 0, 'current-jobs-buried': 0, 'pause-time-left': 0, 'current-waiting': 0, 'current-jobs-ready': 14, 'total-jobs': 14, 'current-watching': 0, 'current-jobs-reserved': 0, 'current-using': 1, 'current-jobs-urgent': 0}
>>> c.reserve()

The exact same code using the default tube works as expected. I can also consume from the tube using telnet. Is this a bug in the library?

The tutorial shows that the tubes method returns a list of values. But the current master returns a string with line breaks, probably meant for printing.

>>> beanstalk.tubes()
'---\n- default\n- foo\n'

I was trying to kick a job today, and it wouldn't work. A little investigation using wireshark, showed that beanstalkd always returned 'KICKED 0'

I found two issues with views.job_kick.

1. beanstalk kick command takes bound as an argument, that is the maximum number of jobs to be kicked in a specific tube. [*]
2. It would seem that using a tube before kicking is necessary.

I can provide a patch to fix this, should be people be interested.

[*] From beanstalkd protocol.txt
The kick command applies only to the currently used tube. It moves jobs into
the ready queue. If there are any buried jobs, it will only kick buried jobs.
Otherwise it will kick delayed jobs. It looks like:

kick \r\n

• is an integer upper bound on the number of jobs to kick. The server
  will kick no more than jobs.

After installing with

pip install --user beanstalkc

Trying this will fails:

~> python
Python 3.9.5 (default, May 14 2021, 00:00:00) 
[GCC 11.1.1 20210428 (Red Hat 11.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import beanstalkc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python3.9/site-packages/beanstalkc.py", line 78
    except CommandFailed, (status, results):
                        ^
SyntaxError: invalid syntax

peek, like reserve, operates on the currently watched tubes.

peek_{ready,delayed,buried} only operate on the currently used tube.

I notice that put(...) only lists ['JOB_TOO_BIG'] as expected errors. Have I missunderstood things or shouldn't it also contain BURIED?

peek_ready gives the first job in the using() tube, not the first job you would get from watching().

The documentation says:
To peek at the same job that would be returned by reserve -- the next ready job -- use peek-ready:

When in reality, peek_ready() returns the next job that will be consumed on the tube you are using, not watching. This completely threw me for a loop.

I wanted to double check and make sure this was intentional and confirmed before I changed the docs.

From https://github.com/earl/beanstalkc/blob/master/beanstalkc.py#L69

self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

It appears beanstalkc only supports IPv4? Can you please add support for IPv6?

Hi,

I'm getting the following stacktrace after I call the delete method on a job:

Traceback (most recent call last):
  File "/var/lib/plateprocessor/processor.py", line 57, in main
    process_beanstalk_messages()
  File "/var/lib/plateprocessor/processor.py", line 30, in process_beanstalk_messages
    job.delete()
  File "/usr/local/lib/python2.7/site-packages/beanstalkc.py", line 272, in delete
    self.conn.delete(self.jid)
  File "/usr/local/lib/python2.7/site-packages/beanstalkc.py", line 229, in delete
    self._interact('delete %d\r\n' % jid, ['DELETED'], ['NOT_FOUND'])
  File "/usr/local/lib/python2.7/site-packages/beanstalkc.py", line 91, in _interact
    raise CommandFailed(command.split()[0], status, results)
CommandFailed: ('delete', 'NOT_FOUND', [])

My code (simplified):

def process_beanstalk_messages():
    beanstalk = beanstalkc.Connection(host=BEANSTALK_HOST, port=BEANSTALK_PORT)
    beanstalk.use(BEANSTALK_ALPRD_QUEUE_NAME)
    beanstalk.watch(BEANSTALK_ALPRD_QUEUE_NAME)

    print("Listening for beanstalk messages..")
    while True:
        job = beanstalk.peek_ready()
        if job is not None:
            logging.info("Received message from beanstalk queue")
            try:
                something()
            except Exception as exception:
                logging.error("Exception ocurred: %s" % exception)
            job.delete()

By default, a beanstalkc.Connection() instance listens to the "default" tube as specified by the beanstalkd protocol (https://github.com/kr/beanstalkd/blob/master/doc/protocol.txt).

It would be useful if there was a constant DEFAULT_TUBE_NAME for that in this library (similar to other defaults) as some application code may wish to use it (e.g., for the purpose of watching other tubes and ignoring that particular default tube).

If this idea seems sensible (even though the rest of beanstalkc.py may not use that constant), I'm happy to create a pull request.

I've got a webapp that's using the kick_job method from HEAD but would prefer to be using a pypi release. Any chance you could push a new version out? Pretty-please? 😀

Send to a tube other than 'default' a delayed job, peek_delayed() won't return that job.
Doing exactly the same with 'default' tube, peek_delayed() will happily return that delayed job.

Could you refactor the constructor of beanstalkc.Connection to support delayed connections? Either make it conditional on a parameter "autoconnect" that defaults to True, or pull the initialization code into a method called "_initialize_connection".

The use case is a service that makes a large number of connections and re-connections to the server. Perhaps we should be maintaining a constant connection, but we are sub-classing beanstalkc.Connection so that it supports the Context Manager syntax (with beanstalkContext: etc.) and would like the connection to happen in the enter and the exit to handle the rollback of jobs that were created on exception. We are overriding the put method to cache the jobs created. We are forced to cut-and-paste the init to avoid connecting to the server on creation of the shared context object.

If you let me know how to submit code, I will submit code for review, if you would prefer. Thank you.

I'm getting a strange stack trace when trying to reserve a job.

File "applications/init/modules/metadata_worker.py", line 31, in <module> job = bs.reserve(timeout=TIMEOUT) File "/usr/local/lib/python2.6/dist-packages/beanstalkc.py", line 131, in reserve ['DEADLINE_SOON', 'TIMED_OUT']) File "/usr/local/lib/python2.6/dist-packages/beanstalkc.py", line 98, in _interact_job jid, size = self._interact(command, expected_ok, expected_err) File "/usr/local/lib/python2.6/dist-packages/beanstalkc.py", line 72, in _interact status, results = self._read_response() File "/usr/local/lib/python2.6/dist-packages/beanstalkc.py", line 81, in _read_response line = SocketError.wrap(self._socket_file.readline) File "/usr/local/lib/python2.6/dist-packages/beanstalkc.py", line 41, in wrap return fn(*args, **kwargs) File "/usr/lib/python2.6/socket.py", line 406, in readline data = self._sock.recv(self._rbufsize)

I don't know what to make of it, as the error name and description are nowhere. Strange thing is, when I run python in the shell (same python as the one running my script above), the reserve thing works intermittently fine. Sometimes even bs.reserve(timeout=0) hangs. Any clues on how to debug this issue?

beanstalkd now has a "put-unique" command:
netsweng/beanstalkd@89e186c

The following happens with beanstalkc from the git repository:

>>> import beanstalkc
>>> beanstalkc.__version__
'0.2.0'
>>> beanstalk = beanstalkc.Connection()
ERROR:root:Failed to load PyYAML, will not parse YAML
>>> beanstalk.put('foo')
4
>>> job = beanstalk.reserve()
>>> print "job.stats() ==", repr(job.stats())
job.stats() == 'id: 3\ntube: default\nstate: reserved\npri: 2147483648\nage: 234\ndelay: 0\nttr: 120\ntime-left: 119\nreserves: 2\ntimeouts: 0\nreleases: 0\nburies: 0\nkicks: 0\n'
>>> job.release()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.5-i386/egg/beanstalkc.py", line 228, in release
TypeError: string indices must be integers

Cannot do nice things like:

with beanstalkc.Connection(host='localhost', port=11300) as beanstalk:
    pass
    #Do stuff

__enter__ and __exit__ are helpful, but were added after v0.4.0

can i use multiple tubes like 'spam' and 'eggs' parallel? so i can reserve job from ['spam'] and from ['eggs'] at same time on same connection.

Hello there :-)
I'm a successful beanstalk user for two years ( using it from go lang programs ) and everything is ok.
Now I tried to wrap a java program into a Python shell, taking one job from a beanstalk queue and runing it into a subprocess.
Everything was ok in tests with only one worker but when I ran 4 of them, I was unpleasantly surprised that all 4 workers started to work simultaneously for the same job, as all of them have reserved & took a bit from the first job in the tube :-(

I mention that the first attempt was to delete the job at the end of a good work but I was forced to immediatly delete the job after release, in order to make just one of 4 workers to do the job.
It's something special that I have to do?
The source is shameless simply as:

try:
    bsk = beanstalkc.Connection(host=server, port=11300, parse_yaml=lambda x: x.split('\n'))
    bsk.watch('registre')
    bsk.use('couchreg')
except:
    print "cannot open connection"
    sys.stdout.flush()
    sys.exit(1)

while 1:
    print time.strftime('%Y-%m-%d %X'), " waiting for work ..."
    sys.stdout.flush()
    job = bsk.reserve(timeout=60)
    if job is not None:
        # I had to delete it immediately
        job.delete()
        try:
            sys.stdout.flush()
            g=os.popen("/bin/bash doJavaWork.sh " + job.body)
            for line in g.readlines():
                stdout.write('%s' % line)
                if 'CASCADE_NEXT' in line:
                    bsk.put(job.body,20,1800)
            print "Finished it"
            sys.stdout.flush()
        except:
            print "bad things"
            sys.stdout.flush()

I'm using with flask like webhook.
When app started Connection really, but after 30 minutes i'm raising error: [Errno 104] Connection reset by peer when use.

Any way to fix connection?
Thanks for reading!
My English not good!

I noticed while doing some somewhat artificial load testing that Connection.close() closes the socket without sending quit, which results in the socket remaining open for a short while in TIME_WAIT. I made a changeset that sends quit first to ensure the socket is closed.

http://github.com/robotadam/beanstalkc

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

It's not quite clear to me how to get beanstalkc to use the CLoader and CDumper libyaml classes instead of the pure Python versions. Could you elaborate on that in the Tutorial, please?

Hi, I notice there may be an socket close bug after viewing the beanstalkc code:

    def close(self):
        """Close connection to server."""
        try:
            self._socket.sendall('quit\r\n')
            self._socket.close()
        except socket.error:
            pass

    def reconnect(self):
        """Re-connect to server."""
        self.close()
        self.connect() 

suppose i made a connection to beanstalkd, but the server close the socket for some reason(server exit,for example). then beanstalkc raise an exception of socket error. so the client code make a reconnect call to establish a new connection to the server. but as you see in the close method, it first sendall('quit'). that will beak. so it does not work around in this case. mybe we could just close the socket,am i right?
thanks:)

Pull request #57 adds python 3 compatibility, but always tries to decode job bodies. Given that arbitrary byte strings may be put in the body, this is suboptimal.

For other protocol messages, only ascii content is accepted, so encoding/decoding automatically is fine.

I propose the following:

• Add an 'encoding' attribute to Connection.__init__, defaulting to sys.getdefaultencoding

For put:

• When you put a bytes objects it's put as-is
• Otherwise it's encoded with the encoding above

And for reseve/peek etc. (_read_body):

• When a body is read:
  • If encoding is None, return a bytes object
  • Otherwise decode with the specified encoding and return a string

If this design is acceptable, I'll rebase and amend the pull request to follow this design.

I get over 6700 exceptions in under a second, and then it works normally again.
What exactly happens is as follows (please, see my implementation of BOSS and WORKER below):

1. The boss and the worker connects
2. I put a job in the tube
3. I receive it
4. Some time later, lots of DeadlineSoon exceptions arrive, very fast
5. After ca. 1s, they stop again, and a new job is received (which I never sent)
6. the situation is normal again, for a while, until we find ourselves again in step 4

My implementation is as follows:

class BeanstalkWorker:
    '''This class receives jobs from beanstalkd, encoded as JSON strings'''
    def __init__ (self, bt_tube, process_data, host = BEANSTALK_HOST, port = BEANSTALK_PORT):
        module_log.info("Connecting to beanstalk host %s, port %d, tube %s" % (host, port, bt_tube))
        self.json_support = JsonSupport()
        self.process_data = process_data
        try:
            self.beanstalk    = beanstalkc.Connection(host=host, port=port)
            self.beanstalk.watch(bt_tube)
        except beanstalkc.SocketError, e:
            module_log.warning("Problems connecting to beanstalk host %s, port %d: %s" % (host, port, e))
            self.beanstalk = None

    def show_stats(self):
        pprint(self.beanstalk.stats())

    def run(self):
        if not self.beanstalk:
            module_log.warning("We have no connection to the beanstalk host")
            return
        stop_requested = False
        while True:
            try:
                job = self.beanstalk.reserve(timeout=1)
            except beanstalkc.DeadlineSoon, e:
                module_log.info("DeadlineSoon: %s" % (e))
                job = None
            if not job:
                module_log.info("No job")
                if stop_requested : break
            else :
                module_log.info("Job found")
                data = self.json_support.Decode(job.body)
                if data:
                    if data.get('management', None) == 'STOP':
                        module_log.info("STOP requested")
                        stop_requested = True
                        job.delete()
                    else :
                        if self.process_data(data):
                            module_log.info("Job completed")
                            job.delete()

class BeanstalkBoss:
    '''This class sends dictionaries encoded as JSON strings to beanstalkd'''
    def __init__ (self, bt_tube, host = BEANSTALK_HOST, port = BEANSTALK_PORT):
        module_log.info("Connecting to beanstalk host %s, port %d, tube %s" % (host, port, bt_tube))
        try:
            self.beanstalk = beanstalkc.Connection(host=host, port=port)
            self.beanstalk.use(bt_tube)
        except beanstalkc.SocketError, e:
            module_log.warning("Problems connecting to beanstalk host %s, port %d: %s" % (host, port, e))
            self.beanstalk = None
        self.json_support = JsonSupport()

    def put(self, data):
        if not self.beanstalk:
            module_log.warning("We have no connection to the beanstalk host")
            return
        json = self.json_support.Encode(data)
        self.beanstalk.put(json)

This is the output:

2012-01-30 16:11:21,829 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:22,833 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:23,837 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:24,841 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:25,845 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:26,848 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:27,852 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:28,856 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:29,861 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:30,865 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:31,520 - beanstalk_support    - INFO       -       - DeadlineSoon: []
2012-01-30 16:11:31,520 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - DeadlineSoon: []
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - DeadlineSoon: []
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - DeadlineSoon: []
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - No job
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - DeadlineSoon: []
2012-01-30 16:11:31,521 - beanstalk_support    - INFO       -       - No job
...

It's a quite weird question but it actually exists in my program. I'm designing a eBay like system, and the logic is that:

When users makes an order, they are required to pay within 30 minutes. Otherwise the order will be closed and the user cannot pay it anymore.
I'm using a delayed job to realize this. Whenever an order is made, a job with 1800 seconds delay is generated. And if the user doesn't pay with 1800s, the job will be consumed to a python script, which marks the order as closed in database and notified the user.

But normally, 80% of the orders will be paid with in 1800s. Thus without these delayed jobs being deleted, 80% of the python computation is meaningless, and at least one mysql query will be executed for each job.

What I want to know, is how to actively delete the specific delayed job stored in an tube in my order logic. Like when the user finished the payment process, the delayed job will be deleted.

Since setup.py include the following line:

git_version = os.popen('git describe --tags --abbrev=6').read().strip()[7:]

it picks up the git version from my projects repo and adds it to the beanstalkc version. This breaks pip if it expecting a specific version because it can no longer match the required version number. For example:

$ ./ve/bin/pip install beanstalkc
$ ./ve/bin/pip freeze | grep beanstalkc
beanstalkc==0.2.0.dev2-g2a8dd2

Which is fine without a required version, but if I have beanstalkc==0.2.0 in a requirements.pip file:

$ ./ve/bin/pip install -r requirements.pip
Downloading/unpacking beanstalkc==0.2.0 (from -r requirements.pip (line 12))
  Running setup.py egg_info for package beanstalkc
  Source in ./ve/build/beanstalkc has the version 0.2.0.dev2-g2a8dd2, which does not match the requirement beanstalkc==0.2.0 (from -r requirements.pip (line 12))
Source in ./ve/build/beanstalkc has version 0.2.0.dev2-g2a8dd2 that conflicts with beanstalkc==0.2.0 (from -r requirements.pip (line 12))

Any interest in merging pooled connection support if I were to implement it?

Thanks

I'm working with beanstalkd on gentoo (emerged from portage).
My problem is, if I use a default tube everything is ok but if I use other tube I can't reserve any job.
I can peek job, stats_tube shows good info but reserve always returns None.
Why so?

When I use beanstalkc with gevent, I hit the error

[E 160217 14:26:01 event:37 f5e11774ac61efb21a5b9e64006c544d:10] This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter object at 0x7fe0e514fc30>>
Traceback (most recent call last):
File "/data/apps/owl/owl/api/event.py", line 35, in create_event
beanstalk.put(job)
File "/data/apps/owl/owl/beanstalk.py", line 58, in put
self._conn.put(json_encode(job), delay=delay)
File "/data/apps/owl/eggs/beanstalkc-0.4.0-py2.7.egg/beanstalkc.py", line 136, in put
['INSERTED'], ['JOB_TOO_BIG','BURIED','DRAINING'])
File "/data/apps/owl/eggs/beanstalkc-0.4.0-py2.7.egg/beanstalkc.py", line 110, in _interact_value
return self._interact(command, expected_ok, expected_err)[0]
File "/data/apps/owl/eggs/beanstalkc-0.4.0-py2.7.egg/beanstalkc.py", line 87, in _interact
status, results = self._read_response()
File "/data/apps/owl/eggs/beanstalkc-0.4.0-py2.7.egg/beanstalkc.py", line 96, in _read_response
line = SocketError.wrap(self._socket_file.readline)
File "/data/apps/owl/eggs/beanstalkc-0.4.0-py2.7.egg/beanstalkc.py", line 41, in wrap
return wrapped_function(_args, *_kwargs)
File "/usr/local/lib/python2.7/socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
File "/data/apps/owl/eggs/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/socket.py", line 394, in recv
self._wait(self._read_event)
File "/data/apps/owl/eggs/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/socket.py", line 294, in _wait
assert watcher.callback is None, 'This socket is already used by another greenlet: %r' % (watcher.callback, )
AssertionError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter object at 0x7fe0e514fc30>>

I think we can add a threading-safe connection pool to solve it, thanks

in out enviroments, something strange happend, the log file looks as flowing:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/taskmanager-0.1.0a19-py2.6.egg/taskmanager/queue/beanstalkd.py", line 186, in delete
    conn.delete(job_id)
  File "/usr/lib/python2.6/site-packages/beanstalkc-0.3.0-py2.6.egg/beanstalkc.py", line 217, in delete
    self._interact('delete %d\r\n' % jid, ['DELETED'], ['NOT_FOUND'])
  File "/usr/lib/python2.6/site-packages/beanstalkc-0.3.0-py2.6.egg/beanstalkc.py", line 85, in _interact
    raise UnexpectedResponse(command.split()[0], status, results)
UnexpectedResponse: ('delete', 'T', []

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/taskmanager-0.1.0a19-py2.6.egg/taskmanager/queue/beanstalkd.py", line 164, in reserve
    job = conn.reserve(timeout)
  File "/usr/lib/python2.6/site-packages/beanstalkc-0.3.0-py2.6.egg/beanstalkc.py", line 141, in reserve
    ['DEADLINE_SOON', 'TIMED_OUT'])
  File "/usr/lib/python2.6/site-packages/beanstalkc-0.3.0-py2.6.egg/beanstalkc.py", line 105, in _interact_job
    jid, size = self._interact(command, expected_ok, expected_err)
  File "/usr/lib/python2.6/site-packages/beanstalkc-0.3.0-py2.6.egg/beanstalkc.py", line 85, in _interact
    raise UnexpectedResponse(command.split()[0], status, results)
UnexpectedResponse: ('reserve-with-timeout', '---', []

i can't figure it out why we get TIMEOUT response when invoke delete command? and why we get --- response when invoke reserve-with-timeout command?

beastalkc.release(...) is defined to take three parameters - one mandatory and two optional. If the optional parameters are not passed, the command WILL fail.

Example:
>>> from beanstalkc import beanstalkc
>>> c=beanstalkc.Connection()
>>> a="gorgonzola!"
>>> c.put(a)
3
>>> j=c.reserve()
>>> j.jid
3
>>> c.release(j.jid)
Traceback (most recent call last):
File "", line 1, in
File "./beanstalkc/beanstalkc.py", line 196, in release
self._interact('release %d %d %d\r\n' % (jid, priority, delay),
TypeError: %d format: a number is required, not NoneType
>>> c.release(j.jid,0,0)
>>> c.delete(j.jid)
>>>