gleicon / restmq

REST/JSON/HTTP based message queue

REAMDE

💬

restmq's Issues

Error running RESTMQ

I installed RestMQ, getting errors when I try to install it.

Unexpected error while writing cache file
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 614, in parseOptions
usage.Options.parseOptions(self, options)
File "/usr/lib/python2.7/site-packages/twisted/python/usage.py", line 261, in parseOptions
for (cmd, short, parser, doc) in self.subCommands:
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 631, in subCommands
for plug in sorted(plugins, key=attrgetter('tapname')):
File "/usr/lib/python2.7/site-packages/twisted/plugin.py", line 209, in getPlugins
allDropins = getCache(package)
--- ---
File "/usr/lib/python2.7/site-packages/twisted/plugin.py", line 181, in getCache
dropinPath.setContent(pickle.dumps(dropinDotCache))
exceptions.AttributeError: 'ZipPath' object has no attribute 'setContent'
Traceback (most recent call last):
File "/usr/bin/twistd", line 14, in
run()
File "/usr/lib/python2.7/site-packages/twisted/scripts/twistd.py", line 27, in run
app.run(runApp, ServerOptions)
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 652, in run
runApp(config)
File "/usr/lib/python2.7/site-packages/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 386, in run
self.application = self.createOrGetApplication()
File "/usr/lib/python2.7/site-packages/twisted/application/app.py", line 446, in createOrGetApplication
ser = plg.makeService(self.config.subOptions)
File "build/bdist.linux-x86_64/egg/twisted/plugins/collectd_plugin.py", line 32, in makeService
File "build/bdist.linux-x86_64/egg/restmq/collectd.py", line 133, in init
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 882, in resource_filename
self, resource_name
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1352, in get_resource_filename
return self._extract_resource(manager, zip_path)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1363, in _extract_resource
zip_stat = self.zipinfo[zip_path]
KeyError: 'restmq/static'

Getting "Cannot load ACL file" error when trying to start bash script

Followed all install instructions and tried to start bash script "restmq_server" and I get an error "Cannot load ACL file":

Traceback (most recent call last):
File "/usr/bin/twistd", line 5, in
pkg_resources.run_script('Twisted==13.1.0', 'twistd')
File "/usr/lib/python2.6/site-packages/distribute-0.6.49-py2.6.egg/pkg_resources.py", line 507, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.6/site-packages/distribute-0.6.49-py2.6.egg/pkg_resources.py", line 1272, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/lib/python2.6/site-packages/Twisted-13.1.0-py2.6-linux-x86_64.egg/EGG-INFO/scripts/twistd", line 14, in
run()
File "/usr/lib/python2.6/site-packages/Twisted-13.1.0-py2.6-linux-x86_64.egg/twisted/scripts/twistd.py", line 27, in run
app.run(runApp, ServerOptions)
File "/usr/lib/python2.6/site-packages/Twisted-13.1.0-py2.6-linux-x86_64.egg/twisted/application/app.py", line 652, in run
runApp(config)
File "/usr/lib/python2.6/site-packages/Twisted-13.1.0-py2.6-linux-x86_64.egg/twisted/scripts/twistd.py", line 23, in runApp
_SomeApplicationRunner(config).run()
File "/usr/lib/python2.6/site-packages/Twisted-13.1.0-py2.6-linux-x86_64.egg/twisted/application/app.py", line 386, in run
self.application = self.createOrGetApplication()
File "/usr/lib/python2.6/site-packages/Twisted-13.1.0-py2.6-linux-x86_64.egg/twisted/application/app.py", line 446, in createOrGetApplication
ser = plg.makeService(self.config.subOptions)
File "/usr/lib/python2.6/site-packages/RestMQ-1.1-py2.6.egg/twisted/plugins/restmq_plugin.py", line 32, in makeService
options["redis-pool"], options["redis-db"]),
File "/usr/lib/python2.6/site-packages/RestMQ-1.1-py2.6.egg/restmq/web.py", line 666, in init
raise RuntimeError("Cannot load ACL file: %s" % e)
RuntimeError: Cannot load ACL file: [Errno 2] No such file or directory: '/opt/restmq/etc/acl.conf'

GET only deleting object-reference, not actual object

I'm using restmq as intermediate queue feeding it data from collectd and then popping and formating data and sending to OpenTSDB.

The docs say that a GET on the /q/<queue> route will return and dequeue the object. I see from the raw network traffic that restmq does RPOP on collectd_data:queue and hence removes the reference to the object. It does not however remove the actual object which will live on in memory forever.

I see in the source that the JSON interface has a take command which should GET and DEL the actual object. But I'm only getting 400 errors on the /queue route.
The docs say that the route is /queue/<queue_name>, which is incorrect. The route is /queue and the queue_name should be supplied in the JSON data.

I use this command to try to get the JSON interface to work without success:

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"cmd": "take","queue": "collectd_data"}' http://localhost:8888/queue

GET to dequeue sounds dangerous

As I will be destroying data with a supposedly read-only verb.

Instead, could use a DELETE on a /head URI or something like that.

data must be utf-8

after I typed "curl -d "queue=test&msg=foobar" http://localhost:8888/", the following error ocurred:
2012-04-29 01:54:10+0800 [HTTPConnection,0,127.0.0.1] ERROR: oper.queue_add('test', 'None') failed: data must be utf-8

Relative path on acl file loading

The file acl.conf is being loaded using relative path, this sucks because I need to place this file in every dir that I need to launch RestMQ. The code should load it using it's absolute path to fix this.

Relative path on restmq while starting the server (restmq_server)

The line that starts the Twisted server is loading RestMQ using it's relative path, because of this you can only run RestMQ from inside it's own folder.

"curl http://localhost:8888/stats" returns 404

Installed everything, started server script, and tried to access port 8888 at my server ip within browser without success. SSH'ed into server and ran "curl http://localhost:8888/stats" and got this response:

<title>404: Not Found</title>404: Not Found

However, posting and getting work with the following commands:
curl -X POST -d "value=foobar" http://localhost:8888/q/test
curl http://localhost:8888/q/test

Ruby example incorrect

I have tried implementing the em-http-request poller, but the problem with it, is that if gets data in chunks from the stream, and in case there's a big json object stored in queue, em-http splits it into several chunks, causing JSON parse errors.

I was not able to find receive buffer setting in the gem itself, so I would appreciate if you had any suggestions.

/usr/local/bin/twistd: Unknown command: restmq

./start_scripts/restmq_server

results in output:

/usr/lib/python2.7/dist-packages/zope/init.py:3: UserWarning: Module twisted was already imported from /usr/lib/python2.7/dist-packages/twisted/init.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
import pkg_resources
Usage: twistd [options]
Options:
--savestats save the Stats object rather than the text output of the
profiler.
-o, --no_save do not save state on shutdown
-e, --encrypted The specified tap/aos file is encrypted.
-n, --nodaemon don't daemonize, don't use default umask of 0077
_cut_
twistd reads a twisted.application.service.Application out of a file and runs
it.
Commands:
dns A domain name server.
ftp An FTP server.
inetd An inetd(8) replacement.
manhole-old An interactive remote debugger service.
portforward A simple port-forwarder.
socks A SOCKSv4 proxy service.
telnet A simple, telnet-based remote debugging service.
web A general-purpose web server which can serve from a
filesystem or application resource.

/usr/local/bin/twistd: Unknown command: restmq

This is on Mint14.

How is this possible?

Transform RestMQ to a Python Package (egg)

I think that RestMQ should be packed into a Python Egg, so it would be so simple to: setup it, instantiate a queue and integrate it in other packages.