histrio / py-couchdb Goto Github PK

The delete_attachment function deletes the file in the database, but does not delete it from the the document instance

In fact I can see in the source client.py#420:
..
if r.status_code < 206:
_doc['_rev'] = d['rev']
return _doc
..

So only the _rev attribute is updated, but the file informations will remain in the _doc['_attachment'] object

The license in the LICENSE file is BSD-3-Clause, but the pyproject.toml has it incorrectly as BSD-4-Clause.

pyproject.toml:license = "BSD-4-Clause"

(BSD-4-Clause was an older version, now rare to see, which required all 'advertising of a provided feature' to identify the license)

After a couple of ours of debugging I discovered a big problem: many functions use dicts (that is a mutable object) as default argument, like params={}, headers={}

In python this have a non-intuitive behaviour:
http://pythonconquerstheuniverse.wordpress.com/category/python-gotchas/

In my code I was calling a function with no params, but the function had it filled due to a previus call.

Please change all them in the pythonic way:
def func(arg=None):
if arg is None: arg = {}

If an _id does not exist the Database.get function doesn't raise NotFound

It gives a dict: {u'reason': u'missing', u'error': u'not_found'}

I think the client would expect an Exception.

Hi,

I'm working on couchapp (design documents) support for this module. It's inspired by erica (https://github.com/benoitc/erica).

For now it's in pycouchdb.app module and CLI tool has signature like this:

$ python -m pycouchdb.app push -h
usage: python -m pycouchdb.app push [-h] [--url COUCHDB_URL] [--db DBNAME]
                                    [--create-db]
                                    [APPDIRS [APPDIRS ...]]

positional arguments:
  APPDIRS            List of directories holding your couchapps. Defaults to
                     all directories found in your current working directory.

optional arguments:
  -h, --help         show this help message and exit
  --url COUCHDB_URL  CouchDB url. Defaults to `http://localhost:5984/`. For
                     basic auth, use e.g.
                     `http://user:password@localhost:5984`.
  --db DBNAME        Database name. Database must exist and you must have
                     sufficient permissions to read/write design documents
                     from/to it. If omitted, name of parent directory will be
                     used.
  --create-db        Create database if it does not exist.

In the end it should have support for operations:

• push - push directory as couchapp
• clone - clone existing couchapp
• create - create couchapp from included template

Would you like this to be part of pycouchdb? If so, do you have any suggestions?

I would expect that not found items not to be present in list.
Failing testcase is on the way.

refs #36

This is an enhance request

It should be great to have the ability to check for updates

http://guide.couchdb.org/draft/notifications.html

With curl the command is:
curl -X GET "http://localhost:5984/db/_changes?feed=continuous&since=1"

With requests:
r= requests.get("http://localhost:5984/db/_changes", params={"feed":"continuous", "since":1}, stream=True)
for ch in r.iter_lines(chunk_size=1): print ch # ch is a str

I don't like chunk_size=1, but I don't know which could be the right size. 512 is too high. The result "ch" have to be jsonized

With couchdb-python this is the solution:
http://stackoverflow.com/questions/7840383/couchdb-python-change-notifications

Note that couchdb-python is able to check the single changes for each item; I don't know if this is a couchdb or a couchdb-python property

Hey. First of all I'd like to thank you on the work on the library.

Now, I installed pycouchdb through PyPI and I was looking for a way to get the _db_updates feed. The PyPI latest version (1.12) does not provide this functionality. Because this functionality is needed for my current project, I went on to create a fork and see how I could implement it myself. I then realized that it is already available on the repo.

My question then would be: is there anything stopping to release a new version with this feature? Missing documentation? Tests? It would be really good to get this released, and I'd be glad to help.

I have a query like this

balance = int(list(db.query('balance/all', descending='true', limit='1'))[0]['value'])

Which as you can see is a lot less cleaner then being able to do something like this

balance = int(db.one('balance/all'))

Or something similiar, I don't know. I just found it a bit annoying to have to do [0] and especially ['value'] to get the value of a specific document like that.

Suggestions?

'save' and "delete_attachment" functions return an updated copy of the original document using the copy.copy function.

But should it be better to use the "copy.deepcopy" function?
Documents can have nested dictionaries; "_attachments" is one of them (and delete_attachment delete the original key), but users can add their own ones.

For my code is not a big problem as far I usually do "doc = function(doc, ..)"

Allow for paginating views queries

When I try to connect to CouchDB over an HTTPS connection, urllib3 complains that it is making an insecure request:

InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html

Unfortunately, the advice given on the above link can't be applied because pycouchdb doesn't expose any interface with Requests or urllib3.

I tried to get around this by patching the urllib3 connection pool classes (per this answer on StackOverflow):

def patch_connection_pools(**constructor_kwargs):
    """
    Override the default parameters of the HTTPConnectionPool and
    HTTPSConnectionPool constructors
    """
    from requests.packages.urllib3 import connectionpool, poolmanager

    def subtype_connection_pool(base):
        class _ConnectionPool(base):
            def __init__(self, *args, **kwargs):
                kwargs.update(constructor_kwargs)
                super().__init__(*args, **kwargs)

        return _ConnectionPool

    poolmanager.pool_classes_by_scheme['http'] = subtype_connection_pool(connectionpool.HTTPConnectionPool)
    poolmanager.pool_classes_by_scheme['https'] = subtype_connection_pool(connectionpool.HTTPSConnectionPool)

patch_connection_pools(maxsize=16, cert_reqs='CERT_REQUIRED', ca_certs='/path/to/my/cacert.pem')

The patching is definitely being called by pycouchdb (verified by putting a print in the patched constructor), but the cert_reqs and ca_certs arguments have no effect on the warning given by urllib3. (I only assume maxsize is being respected.)

I haven't delved through the source of these libraries too deeply, so I might be targeting the wrong place. Is there any other way I can get pycouchdb to do certificate verification?

like the pull request #55 (use OrderedDict) it would also be nice to have an option to get docs sorted by key.

Stackoverflow recursivly sort by key:

def sortOD(od):
    res = OrderedDict()
    for k, v in sorted(od.items()):
        if isinstance(v, dict):
            res[k] = sortOD(v)
        else:
            res[k] = v
     return res

Good Morning,

I just received this error:

py2.7.egg/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'etag'

After some investigation I found it was due to a bad request (wrong couchdb url, I forgot to put https instead of http):

(Pdb) resource.__dict__
{'session': <requests.sessions.Session object at 0x104283bd0>, 'base_url': 'XXX'}
(Pdb) resource.head()
(<Response [400]>, None)

This might be due to: if the result is none it returns the bad request and none: https://github.com/histrio/py-couchdb/blob/master/pycouchdb/resource.py#L101

I know that the mistake is mine but would it be more preferable for the system to throw an exception such as Bad Request (in that case at least)? https://github.com/histrio/py-couchdb/blob/master/pycouchdb/exceptions.py#L32

Many thanks

It appears that replacing urljoin by urllib.parse.urljoin causes a problem when getting a user from couchdb. I have found that it is caused by urllib handling the colon in typical user id e.g. org.couchdb.user:username

>>> from urlparse import urljoin
>>> urljoin('http://example.com', 'org.couchdb.user:username')
'org.couchdb.user:username'

The same applies to python 3.x

This leads to

requests.exceptions.InvalidSchema: No connection adapters were found for 'org.couchdb.user:username'

To me it looks like urllib problem as I would say a colon is an allowed character.

In Python 3 byte / str problem

requests/models
the folowing code tests if isinstance(data, str) or isinstance(data, builtin_str) or hasattr(data, 'read'):

if files:
            (body, content_type) = self._encode_files(files, data)
        else:
            if data:
                body = self._encode_params(data)
                if isinstance(data, str) or isinstance(data, builtin_str) or hasattr(data, 'read'):
                    content_type = None
                else:
                    content_type = 'application/x-www-form-urlencoded'

        self.prepare_content_length(body)

        # Add content-type if it wasn't explicitly provided.
        if (content_type) and (not 'content-type' in self.headers):
            self.headers['Content-Type'] = content_type

the self.headers dict will contain

b'Content-Type' : b'application/json'
'Content-Type' :  'application/x-www-form-urlencoded'

resulting a couchdb protocol error:

{"error":"bad_content_type","reason":"Content-Type must be application/json"}

With the wrong filename get_attachment doesn't raise any error (but gives an str with the json of the error)

Hi @histrio

I hope you can help me:) I struggle with putting this value: [{db_fragmentation, \"50%\"}, {view_fragmentation, \"50%\"}]into the CouchDB config -> invalid JSON.

Do you know how to write/encode \"50%\"? When I remove the \" it is valid JSON, but not a valid configuration value:

pycouchdb.resource.Resource("http://192.168.42.100:5984").put('_config/foo/bar', data="\"[{db_frag, false}]\"")

I tried some combination and played with adding params={"raw":"true"}, so fare without success:(

Do you have an idea?

Thanks a lot, Leif

Hey everyone,

in order to authenticate with a CouchDB server, you need to add the basic authentication to the url itself, where it is extracted afterwards. If I get username and password separately, I need to put in way to much work to cut the url and put it back together including the authentication details. Looking into the code, the authentication is extracted on the second step when creating a new Server connection. Would it be possible to provide the means to supply user and password separately?

Something like

couchdb = pycouchdb.Server(couch_uri, auth=("user", "pass"))

I tried using .all without the include_docs parameter, but the following line
yield wrapper(row['doc'])
references 'doc' which does not exist in the returned json when include_docs is False.

I want to get all design docs from a db:

for doc in db.all(startkey="_design/", endkey="_design0"):

and gets

File "...\pycouchdb\utils.py", line 116, in _encode_view_options
    value = json.encode(value)
AttributeError: 'module' object has no attribute 'encode'

I naively fixed it with the utils.to_json() function

I don't know why, but I've tried to build the docs out of this master repository, and the new replicate() method appears. As far as I can see, it's not on: https://py-couchdb.readthedocs.org/en/latest/api.html#server

I thought that the RTFD build was automagic via the "admin" settings. Do we need to do something special in the commits to get it built?

In client.py#416 I can see:
..
414: r = resource.delete(filename, params={'rev': _doc['_rev']})
415: if r.status_code == 404:
416: raise NotFound("filename {0} not found".format(filename))
..

As far we can -try to- delete a nonexistent file without get any error I think the 416 row is showing the wrong error message.

In fact:
curl -X DELETE http://localhost:5984/composite/3c4d73b523db075f51884e1fea0000af/notexists.pdf?rev=30-88c02febc80225edb13c776e86460b27
{"ok":true,"id":"3c4d73b523db075f51884e1fea0000af","rev":"31-5a690ab5da880a639455af989ed1e895"}

I'm using the couchdb version "1.4.0+build.6187ba5" , I don't know if this is the standard (and not buggy) couchdb behaviour

I was trying to upload a simple design document with the document ID "_design/browse". It failed with

  File "/usr/local/lib/python3.4/dist-packages/pycouchdb/client.py", line 225, in save
    if "rev" in result and result["rev"] is not None:
TypeError: argument of type 'NoneType' is not iterable

Saving other documents (non-design documents) works without problems. Deleting, creating a DB, etc. works too, so it is not a problem related to permissions (I'm connected as admin user).

After further investigation, I found out that the response from the server is a 404 document not found error. I further found out that it was trying to save the document to the URL http://127.0.0.1:8080/couchdb/bsdp_v2/_design%2Fbrowse which I guess reveals the error. I think it should be http://127.0.0.1:8080/couchdb/bsdp_v2/_design/browse, so slashes shouldn't be URL-encoded (at least not in the case of design documents.

I'm using pycouchdb 1.12.

I have a web crawling app that creates a buffer of about 500 documents and bulk saves them to couch. Whenever I'm getting too many page errors, all of a sudden, for some unknown reason, this error pops up. I don't understand why page load errors would interfere with bulk save since I only buffer documents that have been correctly loaded. Any idea why this is happening?

Traceback (most recent call last):
File "fetch.py", line 28, in
fetcher.fetch()
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 31, in fetch
loop.run_until_complete(self.run())
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", line 467, in run_until_complete
return future.result()
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 43, in run
await asyncio.gather(*all_jobs)
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 55, in cnpj_job
self.flush()
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 63, in flush
self.cnpjcouchdb.save_bulk(list(self.couch_buffer))
File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\cnpjcouchdb.py", line 25, in save_bulk
return self.db.save_bulk(docs)
File "C:\Users\danilo\Projects\CNPJReceita\venv\lib\site-packages\pycouchdb\client.py", line 403, in save_bulk
_docs = copy.deepcopy(docs)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 274, in _reconstruct
y = func(*args)
TypeError: init() missing 1 required positional argument: 'history'

I use very big attachments (50mb? 100Mb?..)

Can I suggest to stream the get_attachment result?

def get_attachment(self, doc, filename, stream=False):
..
r = self.resource(doc['_id']).get(filename, stream=stream)
..
if stream is True:
return r
return r.content

Not tested

Pull request incoming.

I am looking for mango type query supoort? LIke: { "selector": { "year": { "$eq": 1988 } }}
If it is supported by py-couchdb, how to use it? If not, Will be supported in the future ?

https://travis-ci.org/histrio/py-couchdb/jobs/443872408

Failed building wheel for biscuits

https://github.com/histrio/py-couchdb/blob/master/setup.py#L10 eventually causes an import of the requests library which can not be installed until the setup function is called and can read the install_requires value.

Just need to duplicate the version value or read from a shared file in the package to get around this problem.

When I upload one file the code looks like:

doc = db.put_attachment(doc, file_object)

Ok, but now I want to update one field:

doc['onefield'] = 'newcontent'
doc = db.save(doc)

et voilà, the previously uploaded file has been deleted!

This happens because "put_attachment" doesn't update the doc['_attachment'] , so if I save it later I will delete any new attachment.

The problem is that "put_attachment" can't fill the "doc" with all the informations: content_type, revpos, digest, length. In fact couchdb doesn't help us: when we put an attachment it responds only with id, rev and status. Nothing more.

Probably is not a good idea to let put_attachment does the couchdb job (calculating all the file informations).

The only solution seems to be to read the document again from the db.

why not providing a DB replication method? something like:

db.replicate_from('http://example.com:5984/database')

or

db.replicate_to('http://example.com:5984/database')

Couchdb supports selectors when listening to changes streams, but you need to be able to POST your selector to the changes stream url.

Hello,

I have an application using pycouchdb library and I found this error on my application

File "pycouchdb\client.py", line 453, in all File "pycouchdb\resource.py", line 113, in get File "pycouchdb\resource.py", line 108, in request File "pycouchdb\resource.py", line 72, in _check_result pycouchdb.exceptions.GenericError: {'error': 'killed', 'reason': '{gen_server,call,[<0.24563.30>,bytes,infinity]}', 'ref': 2244742639}

Can someone help me to understand what is the meaning of this error.

Thank you.

Hi

I don't see the language pull-down menu in Futon
after following the steps in
https://pycouchdb.readthedocs.io/en/latest/views.html

modified the /opt/couchdb/etc/local.ini
to
[query_servers]
python3 = /usr/local/bin/couchpy
systemctl stop couchdb
systemctl start couchdb

Can you please help since I am getting the error
Save failed. Reason: Compilation of the map function in the 'pyview' view failed: Expression does not eval to a function. (def fun(doc): if "name" in doc: yield doc['name'], None)
when I use python in the view code

unicode is not str under python 2.7.X.

Error is:

ERROR: test_quote (__main__.UtilsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 491, in test_quote
    self.assertEqual(couchdb.utils.quote('Š'), '%C5%A0')
  File "/Users/bmihelac/dev/django/tmp/py-couchdb/pycouchdb/utils.py", line 55, in quote
    return _quote(data, safe)
  File "/opt/boxen/homebrew/Cellar/python/2.7.3-boxen2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1268, in quote
    return ''.join(map(quoter, s))
KeyError: u'\u0160'

As fat I need to get an attachemnt to a particular revision I need the "rev" argument to be added to the get_attachment function.

I can propose to add the kwargs (like for many other funtions):

def get_attachment(self, doc, filename, stream=False, **kwargs):
..
r, result = self.resource(doc['_id']).get(filename, stream=stream, params=kwargs)
..

Pycouchdb doesn't need the uuid module as far couchdb provides a good "_id" if it is not present in the document at first save.
The couchdb's _id is guaranteed to be unique in the -single- db.

I see in the "save" function:
..
self.resource(_doc['_id']).put(data=data)
..

that should become similar to the following if "_id" not in doc:
self.resource().post(data=data)

Similar update for save_bulk

Hi @histrio,

thanks for your great work. Could you help me understand the why behind escaping all slashes within path parts (see utils.py:76)?

My expectation was that the code below should work:

# curl -X PUT http://192.168.59.103:5984/_config/couch_httpd_auth/timeout -d '"42"'
resource = Resource("http://192.168.59.103:5984")
resource.put("_config/couch_httpd_auth/timeout", data='"42"')

The code is almost a one to one copy from the CouchDB REST API (see comment), but unfortunately doesn't work. I have to write the following:

resource = Resource("http://192.168.59.103:5984")
resource.put(["_config", "couch_httpd_auth", "timeout"], data='"10"')

which is IMHO not as readable and as intuitive as the above example.

Thanks, Leif

Dear Team,

I tried to install py-couchdb in ubuntu12.04 with CouchDB 1.6.1 and python 2.7 as well as python 3.3 and got the installation errors as given below:

$ pip install pycouchdb
Downloading/unpacking pycouchdb
Downloading pycouchdb-1.13.tar.gz
Running setup.py (path:/tmp/pip_build_otc/pycouchdb/setup.py) egg_info for package pycouchdb
Traceback (most recent call last):
File "", line 17, in
File "/tmp/pip_build_otc/pycouchdb/setup.py", line 10, in
import pycouchdb
File "pycouchdb/init.py", line 10, in
from .client import Server
File "pycouchdb/client.py", line 13, in
from .resource import Resource
File "pycouchdb/resource.py", line 5, in
import requests
ImportError: No module named requests
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 17, in

File "/tmp/pip_build_otc/pycouchdb/setup.py", line 10, in

import pycouchdb

File "pycouchdb/init.py", line 10, in

from .client import Server

File "pycouchdb/client.py", line 13, in

from .resource import Resource

File "pycouchdb/resource.py", line 5, in

import requests

ImportError: No module named requests

Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_otc/pycouchdb
Storing debug log for failure in /tmp/tmpVI3k_S
$

Please advise.

Philip.

Strange thing: Some values containing special characters will not read correctly after getting them back from the database.

I tested the following with requests version 1.1.0, 1.2.3 and 2.0.1 (see below).

Here are two tests I've included in the DatabaseTests class (see my fork):

    def test_special_chars1(self):
        text="Lürem ipsüm."
        self.db.save({"_id": "special1", "text": text})

        doc = self.db.get("special1")
        self.assertEqual(text, doc["text"])

    def test_special_chars2(self):
        text="Mal sehen ob ich früh aufstehen mag."
        self.db.save({"_id": "special2", "text": text})

        doc = self.db.get("special2")
        self.assertEqual(text, doc["text"])

The Result is:

Failure
Expected :'Mal sehen ob ich früh aufstehen mag.'
Actual   :'Mal sehen ob ich frĂźh aufstehen mag.'

For the different requests versions (if it matters):

• 1.1.0 Both tests fail.
• 1.2.3 The second test fails.
• 2.0.1 The second test fails.

This is very strange. I'm pretty sure, that the problem does not occur while saving the document, but while receiving it.

Could anyone help? I'm not even sure it is a problem of py-couchdb or requests or...