inoi / trombi Goto Github PK

I'm seeing this in tornado's AsyncHTTPClient's logging:

root: INFO: POST http://localhost:5984/mydb/_design/mydesign/_view/myview?keys=%5B%5B%22value1%22%2C+%22value2%22%5D%2C+%5B%22value3%22%2C+%22value4%22%5D%5D
root: DEBUG: Re-using existing connection! (#0) with host localhost
root: DEBUG: Connected to localhost (127.0.0.1) port 5984 (#0)
root: DEBUG: > POST /mydb/_design/mydesign/_view/myview?keys=%5B%5B%22value1%22%2C+%22value3%22%5D%2C+%5B%22value3%22%2C+%22value4%22%5D%5D HTTP/1.1
root: DEBUG: > User-Agent: Mozilla/5.0 (compatible; pycurl)
root: DEBUG: > Host: localhost:5984
root: DEBUG: > Accept: */*
root: DEBUG: > Accept-Encoding: gzip,deflate
root: DEBUG: > Content-Type: application/json
root: DEBUG: > Content-Length: 72
root: DEBUG: > 
root: DEBUG: > '{"keys": [["value1", "value2"], ["value3", "value4"]]}'

Seems that the keys parameter is sent both in query string and in the POST body.

When a Document is instantiated, _id is added as the self.id attribute to the object, but not added to self.data. However when retrieving a view, the documents come back as plain dicts, with no self.id attribute and having an _id key.

The net result of this is that there's an inconsistent feel to query results. You must either use Document.raw() or map Document over ViewResults. _id especially is something that must be dealt with on every record, so having to access it two different ways isn't appealing.

My suggestion would be to put all the values in self.data, regardless of whether they start with underscore or not.

I'm seeing tracebacks like this in my application:
...
File "/usr/lib/pymodules/python2.6/trombi/client.py", line 298, in _really_callback
data = json.loads(response.body)
File "/usr/lib/python2.6/json/init.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

I don't know exactly what CouchDB responds, but the response doesn't contain valid JSON anyhow. It could be that CouchDB cannot be connected to an curl is returning 599.

Anyway, Trombi should handle this situation and return its own error object in this case and not just crash.

Document.attach does not check for errors before calling callback

Title says it all: When include_docs=True, row['doc'] should be converted to trombi.Document automatically

To reduce possible problems with closures and argument passing, add self as origin to an error every time it is raised. This way the caller has a easy way to access Database, Server or Document object when an error happens

set([doc_id], data, callback[, attachments=None]) has a really strange signature. It is the only method in trombi, which has first optional positional argument. Arguments parsing is manual (which can be then simplified). The real reason to change it, however, is inability to pass callback keyword argument. This does not allow to use tornado.gen to simplify asynchronous code.
I guess it can't be handled without changing method's signature and breaking existing code. At least the first argument would become required: set(doc_id, data, callback[, attachments=None]).
Please, let me know if there is a clean way to get around this particular issue.

Detailed info: https://groups.google.com/forum/?pli=1#!topic/python-tornado/HwtlAew5usY

So far I'm not sure if this is a bug in Tornado or in my code (though I suspect the latter)

CouchDB provides a list functionality to control the data returned from the views. Trombi needs to support this list functionality too.

Hi!

I have a problem when uploading attachments to CouchDB. I really am not sure if this is a trombi issue, probably it is related to tornado's AsyncHTTPClient or even pycurl/libcurl but I am kind of desperate here as I do not know where to dig further.

Problem is the following. I have a process running Tornado's eventloop and the AsyncHTTPClient. This process is sending and receiving messages to and from other 'workers'. Then the master is uploading stuff into a CouchDB. in about 90% of all cases nothing happens.

But, when I am uploading a file of size 1.3/1.4 MB to CouchDB I am getting this error in my logs:

httpclient.py:354 - Exception in callback <functools.partial object at 0x2e43ec0> - (<type 'exceptions.TypeError'>, TypeError('expected string or buffer',), <traceback object at 0x2e4ca70>)
Traceback (most recent call last):
    File "tornado-1.1.1-py2.6.egg/tornado/httpclient.py", line 349, in _finish
      time_info=time_info))
    File "tornado-1.1.1-py2.6.egg/tornado/stack_context.py", line 128, in wrapped
      callback(*args, **kwargs)
    File "trombi/client.py", line 480, in _really_callback
      data = json.loads(response.body)
    File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
      return _default_decoder.decode(s)
    File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

After some debugging and printing I discovered that pycurl returned a HTTP 599 timeout error. So I increased the default request_timeout from 20 to 60 seconds but to no avail, again the same problem...

Does anyone have a hint on this?

The only real issue with trombi at this point is the not so well error handling. The curl client is closed and you have to restart the whole process in order to get access to CouchDB again...

Best
Daniel

The session handling is pretty borked in the last pull request, which I singlehandedly messed up. Need to fix ASAP.

couchdb-python has a nice API for bulk document updates. I'd like to see a similar API in trombi.

CouchDB supports fetching document attachments without fetching to document itself:

GET /db/doc/attachment_name

Trombi should also support this. For example:

database.get_attachment('doc', 'attachment_name', callback)

Tornado 2.0 is Python 3.2 compatible, so probably we should make trombi to be too. :)

CouchDB 1.1 now servers documents with content-type: multipart/related if attachments=True. Quick fix is to pass Accept: application/json to the server, better fix would be to support the multipart format.