cloudant / python-cloudant

A Python library for Cloudant and CouchDB

⚠️ NO LONGER MAINTAINED ⚠️

This library is end-of-life and no longer supported.

This repository will not be updated. The repository will be kept available in read-only mode.

Please see the Migration Guide for advice about migrating to our replacement library cloudant-python-sdk.

For FAQs and additional information please refer to the Cloudant blog.

Cloudant Python Client

This is the official Cloudant library for Python.

• Installation and Usage
• Getting Started
• API Reference
• Related Documentation
• Development
  • Contributing
  • Test Suite
  • Using in Other Projects
  • License
  • Issues
• Migrating to cloudant-python-sdk library

Installation and Usage

Released versions of this library are hosted on PyPI and can be installed with pip.

In order to install the latest version, execute

pip install cloudant

Getting started

See Getting started (readthedocs.io)

API Reference

See API reference docs (readthedocs.io)

Related Documentation

• Cloudant Python client library docs (readthedocs.io)
• Cloudant documentation
• Cloudant Learning Center
• Tutorial for creating and populating a database on IBM Cloud

Development

See CONTRIBUTING.md

Using in other projects

The preferred approach for using python-cloudant in other projects is to use the PyPI as described above.

Examples in open source projects

Getting Started with Python Flask on IBM Cloud

Movie Recommender Demo:

• Update and check if documents exist
• Connect to Cloudant using 429 backoff with 10 retries

Watson Recipe Bot:

• Use Cloudant Query to find design docs

License

Copyright © 2015 IBM. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Issues

Before opening a new issue please consider the following:

• Only the latest release is supported. If at all possible please try to reproduce the issue using the latest version.
• Please check the existing issues to see if the problem has already been reported. Note that the default search includes only open issues, but it may already have been closed.
• Cloudant customers should contact Cloudant support for urgent issues.
• When opening a new issue here in github please complete the template fully.

Migrating to cloudant-python-sdk library

We have a newly supported Cloudant Python SDK named cloudant-python-sdk. For advice on migrating from this module see MIGRATION.md.

python-cloudant's Issues

On database.get_view_result() group_level is expecting type 'basestring'

Shouldn't this accept an integer since it's going to be set as a numeric value?
cloudant.errors.CloudantArgumentError: Argument group_level not instance of expected type: <type 'basestring'>

Add index management unit tests for design_document and views modules

In support of #40.

Additional indexes being added to design doc handler?

Right now it looks like the ddoc handler can only manage secondary indexes. Are things like Lucene, CQ and Geo coming too?

Fix DesignDocument so that an empty view sub-document is not created

When instantiating a DesignDocument, an empty views sub-document is created by default. If no view is added to the DesignDocument and that DesignDocument is then created/saved to the remote server, that empty views sub-document is written to the DesignDocument. This should not be the case. For example:

with DesignDocument(db, '_design/ddoc001x') as ddoc:
    ddoc['foo'] = 'bar'
print db['_design/ddoc001x']

will output:

{u'_rev': u'1-95205e2e4d80983f11db5f6018adb430', u'_id': u'_design/ddoc001x', u'foo': u'bar', u'views': {}}

This means that the design document saved to the database contains 'views': {} which it should not.

• Fix the DesignDocument.save() method to check for an empty views sub-document and to remove it before saving it.
• One thing to keep in mind is that removing the "views" key from the DesignDocument object could cause unwanted side effects and introduce new bugs, so a poosible solution would be to copy the DesignDocument content and update that instead as part of the save() method and then updating the original DesignDocument with the new "_rev"
• This bug is somewhat related to #66 and should be resolved before or as part of that bug fix.

Add index management support to library

We need to add in index management functionality to library so that users no longer have to use the _index endpoint directly to perform operations on indexes when using this library.

Add Result class validation logic

We need to add unit tests that test Result class functionality against CouchDB and Cloudant database instances. Once these Result unit tests are in place we should have a full suite of unit tests that test this library against database instances. We should then consider removing the mocked unit test versions as they may be redundant and at times ineffective w.r.t. testing of functionality. The removal of the mocked tests may be spun off as a separate issue.

Revisit the need for copyright notice at the top of each .py file

It was mentioned during a PR review that having a copyright statement at the top of each .py file is not really "Pythonic". We chose at the time to keep all of the copyright statements and revisit the topic at a later date, perhaps with legal counsel that is well versed in open source code.

See conversation starting here: #13 (comment)

Fix CouchDatabase.create_document to handle DesignDocuments appropriately

• Currently database.CouchDatabase.create_document() creates a Document regardless of whether the document is a Document or a DesignDocument.
• We need to add logic to ensure that if the "_id" field passed in as part of the data starts with '_design/' then this method should create a DesignDocument object and that DesignDocument object should be attached to the locally cached database dictionary.

Add query support to the python-cloudant library

We need to add in query management functionality to library so that users no longer have to use the _find endpoint directly to perform queries when using this library.

CloudantDatabase doesn't allow for configuring fetch_limit

CloudantDatabase takes in fetch_limit as a parameter but will set the value to 100 regardless of what the class is instantiated with due to this line

Cannot use `Cloudant(username, password, account=account)` as context manager

The following code fails:

# With context manager
with Cloudant(USERNAME, PASSWORD, account=ACCOUNT_NAME) as client:
    session = client.session()
    database = client[DATABASE_NAME]
    document = database[DOC_NAME]
    print "Got document from context manager: ", document

With the following exception:

Traceback (most recent call last):
  File "main.py", line 20, in <module>
    with Cloudant(USERNAME, PASSWORD, account=ACCOUNT_NAME) as client:
AttributeError: __exit__

Looks like the object returned by the Cloudant call doesn't have the required methods for a context manager?

Investigate possible performance issues with client context manager

See #15 (comment)

Populating _id with a url results in document that cannot be deleted

I don't know if this is a bug or feature request. Either way, to repro:

from cloudant.account import CouchDB
import base64
client = CouchDB('admin', 'admin', url='http://127.0.0.1:5984')
client.connect()

try:
    db = client['some_test_db']
except Exception,e:
    db = client.create_database('some_test_db')

unencoded = 'http://www.google.com'

encoded = base64.b64encode(unencoded)

#succeeds
data = {
    '_id': encoded,
    'someKey': 'someValue'
    }
db.create_document(data)
db[encoded]
db[encoded].delete()

#failure on document delete:
data = {
    '_id': unencoded,
    'someKey': 'someValue'
    }
db.create_document(data)
db[unencoded]
db[unencoded].delete()

Fix database __iter__() to wrap Documents correctly

The CouchDatabase.iter() method haa a bug when it comes to wrapping the yielded documents as Document and DesignDocument. Currently, documents are wrapped as Document(self) and DesignDocument(self). They need to be wrapped as Document(self, doc['id']) and DesignDocument(self, doc['id']). The current way creates Document objects without a _document_id attribute therefore without a valid document_url. This leaves these objects useless when it comes to things related to remote processing.

The "for" block at the end of the iter() method should read like this after the fix:

for doc in docs:
    # Wrap the doc dictionary as the appropriate
    # document object before yielding it.
    document = {}
    if doc['id'].startswith('_design/'):
        document = DesignDocument(self, doc['id'])
    else:
        document = Document(self, doc['id'])
    document.update(doc['doc'])
    super(CouchDatabase, self).__setitem__(
        doc['id'],
        document
    )
    yield document

This problem is referenced here:

#32 (comment)

Obviously this needs to be thoroughly tested and additional unit tests added to check these conditions.

Add index management unit tests for database module

In support of #40

Can you connect using an auth token/API key instead of user/password?

Is there a way to pass a base64 auth token to Cloudant() for creating a new client? Right now I only see passing it username and password. It calls the attribute "auth_token" but it doesn't seem to work unless I use the explicit password.

Code I'm trying:

    with Cloudant(config['cloudant_user'],config['cloudant_auth'], account=config['cloudant_user']) as client:

Reponse:

Traceback (most recent call last):
  File "./dirscan.py", line 818, in <module>
    main(sys.argv[1:])
  File "./dirscan.py", line 152, in main
    load_config(config['passed_config_file'])
  File "./dirscan.py", line 390, in load_config
    with Cloudant(config['cloudant_user'], config['cloudant_auth'], account=config['cloudant_user']) as client:
AttributeError: __exit__

Related question:
I see that I can generate an API key pair, but I don't see any calls in the API to set what permissions those keys have. (Such as setting their permissions on a specific database once they're created.) Am I just missing it?

error message for invalid key name on update could be more informative

I should know better, but I happened to let one of these slip through. A 500 error is a little more raw than I think desirable here:

pip freeze informs:
cloudant==2.0.0a4

Traceback (most recent call last):
File "", line 1, in
File "tests_couchdb.py", line 144, in test_br
my_document.save()
File "/usr/lib/python2.7/site-packages/cloudant/document.py", line 170, in save
put_resp.raise_for_status()
File "/usr/lib/python2.7/site-packages/requests/models.py", line 837, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://127.0.0.1:5984/my_database/julia30

Example code:
from cloudant.account import CouchDB
client = CouchDB('admin', 'admin', url='http://127.0.0.1:5984')
client.connect()
my_database = client.create_database('my_database')
my_database = client['my_database']

data = {
'_id': 'julia30', # Setting _id is optional
'name': 'Julia',
'age': 30,
'pets': ['cat', 'dog', 'frog']
}

Create a document using the Database API

my_document = my_database.create_document(data)

from cloudant.document import Document

First retrieve the document

my_document = my_database['julia30']

Update the document content

This can be done as you would any other dictionary

my_document['name'] = 'Jules'
my_document['age'] = 6

my_document['_iShouldKnowBetter'] = True

You must save the document in order to update it on the database

my_document.save()

It looks like a solution to this might eventually be wired up via:
https://github.com/cloudant/python-cloudant/blob/master/src/cloudant/errors.py ?

I wasn't (quickly) able to determine if I might be able to help here.

Fix Cloudant._usage_endpoint() to only accept both year and month or neither

In the account module, the Cloudant._usage_endpoint() method should only accept either both a year and a month or nothing. Currently an exception is raised when a month is provided but a year is not. We need to update the logic so that an exception is raised if either year or month is missing and of course to not raise an exception if both or neither are provided.

The following curl statements confirm this:

acurl -X GET https://ACCT.cloudant.com/_api/v2/bill/2015
acurl -X GET https://ACCT.cloudant.com/_api/v2/usage/data_volume/2015
acurl -X GET https://ACCT.cloudant.com/_api/v2/usage/requests/2015

The returned response for each was:

{
  "error": "The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again."
}

Use assertRaises in unit tests to test exceptions

Currently we are using the pattern:

try:
    do_something()
    self.assertFail('The above line should fail')
except SomeException, err:
    self.assertEqual(str(err), 'expected error message.')

to check that an exception or error is raised as expected in our unit tests.

Instead we should be using:

with self.assertRaises(SomeException) as cm:
    do_something()
err = cm.exception
self.assertEqual(str(err), 'expected error message.')

This is the case throughout the unit test code base and should be fixed everywhere.

Clean up and automate integration tests

The tests found in the tests/integration folder need to be cleaned up and made to execute against both CouchDB and Cloudant instances. These tests should also be added to the Travis CI process much in the same way that the tests found in tests/unit/db are currently being executed as prt of every Travis build.

Unable to fetch document

I am saving documents in the cloudant with url as the value of _id field. I am able to save the documents successfully but while retrieving its giving me key error.
Example:
client = Cloudant(USERNAME, PASSWORD,url=cloudantURL)
client.connect()
data={'_id':'https://www.mssdfs.com'}
db = client['dbname']
db.create_document(data)
name='https://www.mssdfs.com'
doc = db[name]
if doc.exists():
print True
else:
print False

this gives me key error.

Documentation: connection pooling

One of the failings of the currently python library is that it doesn't describe how to use connection pooling.

This ticket is to track documentation that is quite important to developers around setting up and configuring connection pooling for this library. For example:

1. information on instantiating the library so that it can be shared, e.g. using a singleton
2. calling out which methods are threadsafe and which aren't
3. describe how to configure connection pooling
4. describe how the library authenticates to Cloudant (using cookie or basic authentication)
5. describe if session expiry gets handled automatically by the library [1]. if not, what does the library consumer need to do to handle this?
6. how to use the library in common python application server environments [2] so that connection pooling can be configured at the app server level

---

[1] this assumes the library uses cookie authentication. The docs state that the cookie can be used until it expires.
[2] e.g. django, flask, gunicorn

pip install not working from Project Jupyter

Moving this to its own issue.

See: #49 (comment)

Change view(ddoc,view, keys=[key1,key2]) to use JSON data for keys instead of encoding in URL

Right now, the number of keys you can get from a view in the library is limited by the maximum length of the URL, which is woefully small. Can this be changed to execute a GET or POST with -d {keys: [key1,key2...]} ?

Cannot update _security document using Cloudant python

You cannot update a _security document using python-cloudant. If you run the following script:

from cloudant.account import Cloudant

client = Cloudant(USERNAME, PASSWORD, account=ACCOUNT_NAME)
client.connect()
session = client.session()
database = client[DATABASE_NAME]
security_document = database[DOC_NAME]
print "Got security_document: ", security_document

# Update the security_document with the API key permissions
security_document[u'cloudant'][u'bouninamendouldnimendepa'] = [u'_reader']

security_document.save()

client.disconnect()

You get a problem because _security documents do not have _rev fields (they are not versioned in the same way):

Traceback (most recent call last):
  File "main.py", line 23, in <module>
    security_document.save()
  File "/Users/mike/scratch/python-cloudant/issue-49/lib/python2.7/site-packages/cloudant/document.py", line 172, in save
    super(Document, self).__setitem__('_rev', data['rev'])
KeyError: 'rev'

I should've spotted this issue; it's probably an issue elsewhere too.

Consider one location for library version number

In order to reduce/remove the possibility of discrepancy between the __version__ string found in the library and the version of the library pushed to PyPI, we should place the version number in one location.

Based upon what I've read, there are three ways that I recommend we do this.

1. Put the version in setup.py and then in __init__.py we check for the version number via the package's metadata. That is, we add something like this to __init__.py (this comes straight from SO without a license, so we can't use this code verbatim)

from pkg_resources import get_distribution, DistributionNotFound
import os.path

try:
    _dist = get_distribution('cloudant')
    # Normalize case for Windows systems
    dist_loc = os.path.normcase(_dist.location)
    here = os.path.normcase(__file__)
    if not here.startswith(os.path.join(dist_loc, 'cloudant')):
        # not installed, but there is another version that *is*
        raise DistributionNotFound
except DistributionNotFound:
    __version__ = 'Please install this project with setup.py'
else:
    __version__ = _dist.version

1. Create a VERSION file that looks like

2.0.0b1

and then read it in __init__.py

1. Create a _version.py file that provides the looks like

__version_info__ = (2, 0, '0b1')
__version__ = '.'.join(__version__info)

We could also consider adding 'description', 'license', etc. to _version.py. It could be hard-coded here, and read by setup.py during the build, yet __init__.py would also read the version/info/etc from _version.py without having to resort to the installed package info (which, I suppose, would only be useful if somebody is using this library from the repository without actually installing it in their environment).

Remove make_result from Query and View

The make_result method in View and Query classes is redundant and confusing. It should be removed. A Result/QueryResult object can be stood up that essentially would perform the same functionality. Likewise the database convenience methods of get_view_results and get_query_results also perform the same functionality.

• database module
  • change get_view_results to standup a new Result object passing it **kwargs.
  • change get_query_results to standup a new QueryResult object passing it **kwargs.
• result module
  • Move parameter docs from make_result methods to Result/QueryResult class docstring.
  • Remove references to make_result in comments.
• view module
  • Remove make_result.
  • Refactor custom_result to use either the result attribute on View or simply standup a new Result object passing it **kwargs.
  • Add result attribute to the class docstring.
• query module
  • Remove make_result.
  • Refactor custom_result to use either the result attribute on Query or simply standup a new QueryResult object passing it **kwargs.
    • Add result attribute to the class docstring.
• Tests
  • Ensure current tests still work.
  • Remove make_result tests.

Add unit tests for indexes module

In support of #40.

Updating document

I have tried your documents and gone through the source code. For some reason a request is never made to save() the document.

In your sample code:
there is no mention of _rev but I added it from your API reference.
You import Document but never use it.

Whatever is happening it isnt working. Having wasted a few days on the Cloudant I am ready to move on simply because I don't want to commit more time here. Had I known earlier about this like when I started with Elasticsearch I might have just use plain ol' requests library to construct my queries.

Can someone atleast take a look at prove me wrong and foolish

Add dbcopy to a view?

Is this possible in python-cloudant? In cloudant-python you would just pass the whole view dictionary which could have dbcopy as a field.

For example in cloudant-python you could do something like this.

view = {
    "page_id_tester": {
        "map": "function (doc) {\n if(doc.alarm_id) {\nemit(doc.alarm_id, "
               "doc); }}",
        "reduce": "function (keys, values, rereduce) { var latest = ''; var "
                  "ts = 0; var i = 0; for(i = 0; i < values.length; i++){ "
                  "if(values[i]['timestamp'] > ts){ ts = values[i]["
                  "'timestamp']; latest = values[i];}}return latest;}",
        "dbcopy": "page-view-db"
    }
}

response = design_doc.put(params={
        "_id": "_design/%s" % name,
        "views": view
    })

But if i'm reading the documentation correctly, python-cloudant restricts view creation in a design document by allowing you to pass in map and reduce strings. So is it not currently possible, or am I just missing something(totally possible)?

Update documentation in support of Cloudant Query support

Exception handling

We need to firm up exception handling throughout the library.

• Exception handling throughout needs to be revisited.
• But specifically, I want to call out the way the library (generally) behaves when a user is used to connect to an account that they do not own.

Revisit the need for a MANIFEST.in file

do we need the MANIFEST.in file?

I don't think we do. It doesn't seem to ship the 'requirement.txt' file (I think because we don't include it in the setup.py file

Use configuration file for tests targeting a database

See PR 26 comment: #26 (diff)

Ref: https://docs.python.org/2/library/configparser.html

Optimize performance

We need to consider providing a way for the developer to optimize some of the functionality provided by the library.

• One possible thing to consider is to add a "refresh" local cache attribute to the account, database, and document objects. This will allow the developer to have more control over the library objects that they use. Right now account, database, and document extend dict so with every operation performed to an object that updates it remotely raises two compelling questions:
  • Do you automatically refresh the local cached version for consistency/clarity purposes??
  • Do you not automatically refresh the local cached version for performance reasons??
• Right now, each object class (account, database, document) and even each method within the class seem to handle these two questions inconsistently.
• Another consideration that belongs as part of this case and that is somewhat related to the bullet above is the way in which attachments are handled via a document object. Specifically, when adding or updating attachments we currently perform an extra fetch to refresh the document local dictionary. It would be beneficial if there was a way to remove this last fetch and instead refresh the local document dictionary based on response content and headers, if possible.
  • See this related PR discussion: https://github.com/cloudant/python-cloudant/pull/28/files#r44277879

Add support for CouchDB Admin Party mode

Currently you must have security set up in CouchDB in order to use the lib with CouchDB. Since often CouchDB is set up with "Admin Party" privileges it would be good for the lib to support that as well.

Combine get_view_result and get_view_raw_result into one method

We should merge the database method get_view_raw_result into get_view_result and pass in a parameter of raw_result=False as default to replicate the behavior we recently implemented for query in the database convenience method get_query_result. See https://github.com/cloudant/python-cloudant/blob/51282-add-query-support/src/cloudant/database.py#L795

401 not authorized error when trying to get information about a database

I want to get the size consumed by a database and as there is no function in the library I tried getting this information as follows:

client = Cloudant(USERNAME, PASSWORD,url=cloudantURL)
client.connect()
r=requests.Session()
response=''

url='https://username.cloudant.com/tweetdb'
data={'password':'########'}
response=r.post(url=url,data=data)
print response
client.disconnect()
This is throwing 401 error. Do I need to implement it using curl in python or is there any function in the library for fetching information of a database?

Remove references to Cloudant in CouchDB classes

In various parts of the library, we have Couch* classes, but they have attributes with names like 'cloudant_url'. This doesn't make sense and we should change these names.

This issue seems to be ubiquitous throughout (found in accounts.py, document.py, database.py, result.py, replicator.py) the library and we may need to create subcases for this task. Essentially all of the modules suffer from this either in the code or the documentation.

There should be some priority for this so that we don't let this bleed out into the API for too long.

Plans for Python 3 support

Could you specify in the readme if you have any plans for Python 3 support in the (near) future?.

Perhaps it would make sense to start a list with items to fix for Py3 support such as the use of basestring and type.NoneType.

Fix pylint errors after switching to pylint 1.5.0

As part of the Travis CI build the default version of pylint is now 1.5.0. This switch from 1.4.4 appears to have additional code analysis restrictions that are causing our pylint test that runs as part of the build to return less than a 10 out of 10 which causes the entire build to fail.

• We need to refactor the library code based on the pylint suggestions so that the code base returns 10 out of 10 again using pylint 1.5.0.
• Currently the pylint test is being skipped so that the build does not fail.

Add Basic authentication support to setting up a client session

See #49

create_replication source format

I had little success using create_replication as it is currently set up. I noticed that the format of the source data was different than the format used by the web UI. The current create_replication used the following format for source:

"source": {
"url": "",
"headers": {
"Authorization": ""
}

while the web UI used:

"source": "" (with the appropriate url here)

I made the changes to my code to match the ui and the request succeeded. Is there any known reason as to why the current format would not work?

Don't make the fields argument required for a Query

Currently a python-cloudant query requires the user to pass in a list of fields in the fields parameter, but Cloudant does not require this parameter to be passed. Cloudant will default to returning all fields of a document if fields is not specified. Could python-cloudant be changed to mimic this behavior?

https://github.com/cloudant/python-cloudant/blob/master/src/cloudant/query.py#L185

create_replication target

I have ran into multiple issues while using create_replication. When passing create_replication two couchDB objects, the requests consistently failed. I noticed that the target url was formatted differently than the web UI. create_replication called "database_url" on the database object and it was formatted like so:

user:[email protected]/my_sample_db

while the web UI formatted the url without the pre-appended user:password@:

user.cloudant.com/my_sample_db

I resolved the issue by passing in my own object with the url formatted like the web UI. Is there any reason as to why this would have made a difference?

Expand on development standards in CONTRIBUTING.rst

We need to expand on coding standards, use of pylint, adding/running tests and so on for developers/contributors of this library.

Investigate Feed implementation

• We need to investigate how the Feed class is implemented. Specifically, how a continuous feed works. It appears that the endpoint parameters (for a continuous feed) are not being used to manage how a _changes or a _db_updates call is handled.
• Got feedback from user of 0.5.9 cloudant-python lib.
  • He uses the changes feed and saw that the changes feed in this library doesn't function the same as before. We need to ensure that we are providing an optimal feed solution.

See: cloudant-labs/cloudant-python#31 and cloudant-labs/cloudant-python#33

Change account module to client module

Consider changing the module name from account.py to clients.py. clients.py seems to make more sense since the module contains the CouchDB and Cloudant classes that dictate which client connection to perform. While the notion of account is valid with Cloudant, it is less so clear with CouchDB.

Release process

This lib should have a documented release strategy and process such that releases can be uploaded to pypi.

Add Test Suite content to README

Minimum add instructions on how to run the tests, although that could be in CONTRIBUTING.