basho / riak-python-client

The Riak client for Python.

Python Client for Riak

Build Status

Documentation

Documentation for the Riak Python Client Library is available here.

Documentation for Riak is available here.

Repository Cloning

NOTE: please clone this repository using the --recursive argument to git clone or follow the clone with git submodule update --init. This repository uses two submodules.

Installation

The recommended versions of Python for use with this client are Python 2.7.8 (or greater, 2.7.11 as of 2016-06-21), 3.3.x, 3.4.x and 3.5.x. The latest version from each series should be preferred. Older versions of the Python 2.7.X and 3.X series should be used with caution as they are not covered by integration tests.

Riak TS (Timeseries)

You must use version 2.7.11, 3.4.4 or 3.5.1 (or greater within a version series). Otherwise you will be affected by this Python bug.

From Source

python setup.py install

There are additional dependencies on Python packages setuptools and protobuf.

From PyPI

Official packages are signed and published to PyPI.

To install from PyPI directly you can use pip.

pip install riak

Testing

Unit Tests

Unit tests will be executed via tox if it is in your PATH, otherwise by the python2 and (if available), python3 executables:

make unit-test

Integration Tests

You have two options to run Riak locally - either build from source, or use a pre-installed Riak package.

Source

To setup the default test configuration, build a Riak node from a clone of github.com/basho/riak:

# check out latest release tag
git checkout riak-2.1.4
make locked-deps
make rel

Source build documentation.

When building from source, the protocol buffers port will be 8087 and HTTP will be 8098.

Package

Install using your platform's package manager (docs)

When installing from a package, the protocol buffers port will be 8087 and HTTP will be 8098.

Running Integration Tests

• Ensure you've initialized this repo's submodules:

git submodule update --init

• Run the following:

./tools/setup-riak
make integration-test

Contributors

• Andrew Thompson
• Andy Gross
• Armon Dadgar
• Brett Hazen
• Brett Hoerner
• Brian Roach
• Bryan Fink
• Daniel Lindsley
• Daniel Néri
• Daniel Reverri
• Dan Root
• David Basden
• David Delassus
• David Koblas
• Dmitry Rozhkov
• Eric Florenzano
• Eric Moritz
• Filip de Waard
• Gilles Devaux
• Greg Nelson
• Gregory Burd
• Greg Stein
• Ian Plosker
• Jayson Baird
• Jeffrey Massung
• Jon Meredith
• Josip Lisec
• Justin Sheehy
• Kevin Smith
• Luke Bakken
• Mark Erdmann
• Mark Phillips
• Mathias Meyer
• Matt Heitzenroder
• Matt Lohier
• Mikhail Sobolev
• Reid Draper
• Russell Brown
• Rusty Klophaus
• Rusty Klophaus
• Scott Lystig Fritchie
• Sean Cribbs
• Shuhao Wu
• Silas Sewell
• Socrates Lee
• Soren Hansen
• Sreejith Kesavan
• Timothée Peignier
• tobixx
• Tin Tvrtković
• Vitaly Shestovskiy
• William Kral
• Yasser Souri

riak-python-client's Issues

get_pw/set_pw broken in riak.client

See https://github.com/basho/riak-python-client/blob/8f012cb2e6d39f2f91fcdf6e1ea33e2179bc7668/riak/client.py#L207-225.

large numbers of Links result in poorly formatted HTTP requests

The python client uses a multidict to manage headers. When a header's length is longer than 8192 (when len(header name) + len(header content) + len("\r\n") > 8192) this code will split that header value into multiple values for the same key managed by an instance of multidict. When httplib encodes that multidict it results in:

Link: [' </riak/F.Sn2hPQ/s001>; riaktag="social", ... ']

when it should have been:

Link: </riak/F.Sn2hPQ/s001>; riaktag="social", ...

without the python-style syntax for an array, the: [' ']

This issue is only triggered when there are a large number of links and you are using the http transport.

bucket.get(None).exists() always returns True

Here's the source code to reproduce:

import riak

c = riak.RiakClient()
b = c.bucket("some_bucket_thats_never_used")
print b.get(None).exists()
b.get(None).delete()

Here's the output:

True
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    b.get(None).delete()
  File "/usr/local/lib/python2.7/dist-packages/riak-1.4.0-py2.7.egg/riak/riak_object.py", line 482, in delete
    Result = t.delete(self, rw=rw, r=r, w=w, dw=dw, pr=pr, pw=pw)
  File "/usr/local/lib/python2.7/dist-packages/riak-1.4.0-py2.7.egg/riak/transports/http.py", line 165, in delete
    self.check_http_code(response, [204, 404])
  File "/usr/local/lib/python2.7/dist-packages/riak-1.4.0-py2.7.egg/riak/transports/http.py", line 253, in check_http_code
    raise Exception(m)
Exception: Expected status [204, 404], received 405 : 

Bundled riakclient.proto is out of sync with riak-erlang-client master

I was having issues running the tests for the python client and I tracked it down to the fact that the riakclient.proto file has changed since Riak's latest stable release.

This affects anyone following along with the Riak Fast Track who goes on to try out the python client.

I was able to fix it by replacing riak/transports/riakclient.proto with:

https://github.com/basho/riak-erlang-client/blob/master/src/riakclient.proto

No clear way to populate multiple host/port pairs?

Mostly leaving this as a bookmark to add a patch later, but I noticed that version "2" of the transports accept a connection manager. However, for backwards compatibility, I assume, the RiakClient only accepts host/port.

This brings up the obvious issue that if I wanted to provide multiple host/port pairs to the client, I have to know the internals as far as I can tell.

The other less obvious behavior is that, for example, I use a proxy in front of Riak and I suspect that's not too uncommon. The current connection stuff loops through available connections and then tries a different connection if the current one gets an HTTP reset (which is, I think, a little conservative) but there doesn't seem to be much support for retrying the same host/port. A workaround, obviously, is adding the same host/port more than once. As I said; not obvious, necessarily.

I think the fix is probably something like adding another kwarg to RiakClient's __init__ method like "hostports=None" and allowing a iterable to fill that slot. I don't know if passing in a cm is a good idea, just because it seems like an implementation detail that could change.

Map/Reduce frequently throws riak.RiakError: 'Socket returned short packet length 3 - expected 4'

Last part of the stacktrace:

File "/usr/lib/python2.7/site-packages/riak-1.4.1-py2.7.egg/riak/mapreduce.py", line 230, in run
result = t.mapred(self._inputs, query, timeout)
File "/usr/lib/python2.7/site-packages/riak-1.4.1-py2.7.egg/riak/transports/pbc.py", line 421, in mapred
_handle_response)
File "/usr/lib/python2.7/site-packages/riak-1.4.1-py2.7.egg/riak/transports/pbc.py", line 454, in send_msg_multi
msg_code, resp = self.recv_msg(conn, expect)
File "/usr/lib/python2.7/site-packages/riak-1.4.1-py2.7.egg/riak/transports/pbc.py", line 489, in recv_msg
self.recv_pkt(conn)
File "/usr/lib/python2.7/site-packages/riak-1.4.1-py2.7.egg/riak/transports/pbc.py", line 543, in recv_pkt
% len(nmsglen))
riak.RiakError: 'Socket returned short packet length 3 - expected 4'

It seems to fix the issue after changing the function recv_pkg in riak/transports/pbc.py,

from

    def recv_pkt(self, conn):
        nmsglen = conn.recv(4)
        if len(nmsglen) != 4:
            raise RiakError("Socket returned short packet length %d - expected 4"
                            % len(nmsglen))
        msglen, = struct.unpack('!i', nmsglen)

to

    def recv_pkt(self, conn):
        nmsglen = ''
        while len(nmsglen) < 4:
            x = conn.recv(4 - len(nmsglen))
            if not x:
                break
            nmsglen += x
        #nmsglen = conn.recv(4)
        if len(nmsglen) != 4:
            raise RiakError("Socket returned short packet length %d - expected 4"
                            % len(nmsglen))
        msglen, = struct.unpack('!i', nmsglen)

BTW, I am using riak-python-client 1.4.1

MapReduce doesn't accept non-unicode inputs

When given an ASCII input, MapReduce should determine the correct way to encode the input to unicode so that an exception is not raised when generating the JSON query format.

Add a method to drop all values for an index

Currently with 2i we can remove an (index, value) tuple. It would be nice to be able to drop all of the values for a particular index.

Client incorrectly rejects keys, buckets that are of the unicode datatype

This is problematic because Python's JSON libraries return unicode objects. I ran into this issue when trying to marshall objects derived from a Link-only M/R job, but I imagine this will show up elsewhere as well. My attempt at a patch is in a9fea18 .

TestServer doesn't work for installed library

TestServer only works if run from a VCS checkout, not after "setup.py install". This is due to erl_src being outside the riak namespace. I suggest erl_src gets moved under riak so that it'll automatically get included when install.

Solr support?

Where is the Solr support?

The indexing looks nice and paging support seems excellent for certain features.

TestServer ignores tmp_dir kwarg

TestServer.init takes a tmp_dir argument, but it ignores its value.

special characters in links are not unescaped when using http transport

to_link_header properly uses urllib to escape HTTP's special characters using their percent-equivalent (https://github.com/basho/riak-python-client/blob/master/riak/transports/http.py#L310) however it does not decode links in parse_links (https://github.com/basho/riak-python-client/blob/master/riak/transports/http.py#L331). If a link is stored ("%" -> "%25"), read ("%25"), then stored again ("%25" -> "%2525") the link will become corrupted and fail when used in later processing. This will also cause links to continue to grow in length until they exceed the 8192 - 8 max length which may lead to other problems as well.

Should 2i query run reduce_identity?

The 2i search guidance in the tutorial points at using a command like this in my client code:

client.index( .... ).run()

This appears to run the query, and then prompt a reduce_identity phase - and then convert all the bucket/key pairs into links.

In my use case I'm only interested in returning keys (as strings), and those keys are then filtered locally (I have low cardinality items loaded onto the keys to act like inline fields, and also may wish to locally intersect key sets from different queries). So I can bypass the whole reduce and link conversion phase by doing this:

_inputs = { .... }
transport = client.get_transport()
results = transport.mapred(_inputs, [])

The difference in response time I get when comparing these two methods is significant - especially as the number of keys returned increases, with seconds being saved as the result sets go into tens of thousands. So that leads me to query:

Is running 2i searches through this reduce identity and link conversion the correct default behaviour?
Should there be an option within the client to return keys only and bypass this?

Some of the comparison results below from my environment (these differences are consistent across multiple runs) - timings in seconds:

Search to be run: ['hFLaGC_yOBFamilyName_bin', '02_1973SMITH']
Number of results - Raw Query: 6812
Query response time - Raw Query: 0.565
Number of results - Standard 2i Query: 6812
Query response time - Standard 2i Query: 1.246

Search to be run: ['hFLaGC_yOBFamilyName_bin', '12_1973SMITH']
Number of results - Raw Query: 53
Query response time - Raw Query: 0.043
Number of results - Standard 2i Query: 53
Query response time - Standard 2i Query: 0.076

Search to be run: ['hFLaGC_yOBFamilyName_bin', '01_1973CO+', '01_1973CO~']
Number of results - Raw Query: 7366
Query response time - Raw Query: 0.571
Number of results - Standard 2i Query: 7366
Query response time - Standard 2i Query: 1.234

Search to be run: ['hFLaGC_yOBFamilyName_bin', '01_1973B+', '01_1973C~']
Number of results - Raw Query: 47356
Query response time - Raw Query: 3.345
Number of results - Standard 2i Query: 47356
Query response time - Standard 2i Query: 13.989

Search to be run: ['hFLaGC_F2SparseGivenName_bin', '02_SMNAYELI']
Number of results - Raw Query: 426
Query response time - Raw Query: 0.1
Number of results - Standard 2i Query: 426
Query response time - Standard 2i Query: 0.126

test_generate_key fails

Latest Homebrew version of Riak used.

$ SKIP_LUWAK=1 SKIP_SEARCH=1 python riak/tests/test_all.py 
ss......ssss....F..........................s....ssss.....ss....ssssssssssssssssssssssssssssssssssssssssssssss.......
======================================================================
FAIL: test_generate_key (__main__.RiakHttpTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "riak/tests/test_all.py", line 935, in test_generate_key
    self.assertEqual(len(bucket.get_keys()), 1)
AssertionError: 2 != 1

----------------------------------------------------------------------
Ran 116 tests in 1.417s

FAILED (failures=1, skipped=59)

Test method:

def test_generate_key(self):
    # Ensure that Riak generates a random key when
    # the key passed to bucket.new() is None.
    bucket = self.client.bucket('random_key_bucket')
    for key in bucket.get_keys():
        bucket.get(str(key)).delete()
    o = bucket.new(None, data={})
    self.assertIsNone(o.get_key())
    o.store()
    self.assertIsNotNone(o.get_key())
    self.assertNotIn('/', o.get_key())
    self.assertEqual(len(bucket.get_keys()), 1)

Python client unit tests suffers 4 failures against riak-search_0.14.2-1_i386.deb

Moved from https://issues.basho.com/show_bug.cgi?id=1162, reported by @slfritchie

Using the riak-python-client as of commit 699ae7b, there are four Search-related failures. I'll attach the entire script output for reference.

======================================================================
FAIL: test_solr_search (riak.tests.test_all.RiakHttpTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fritchie/b/src/riak-python-client/riak/tests/test_all.py", line
857, in test_solr_search
    self.assertEquals(1, len(results["response"]["docs"]))
AssertionError: 1 != 0

======================================================================
FAIL: test_solr_search_from_bucket
(riak.tests.test_all.RiakHttpTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fritchie/b/src/riak-python-client/riak/tests/test_all.py", line
829, in test_solr_search_from_bucket
    self.assertEquals(1, len(results["response"]["docs"]))
AssertionError: 1 != 0

======================================================================
FAIL: test_solr_search_with_params
(riak.tests.test_all.RiakHttpTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fritchie/b/src/riak-python-client/riak/tests/test_all.py", line
849, in test_solr_search_with_params
    self.assertEquals(1, len(list(result.iter("doc"))))
AssertionError: 1 != 0

======================================================================
FAIL: test_solr_search_with_params_from_bucket
(riak.tests.test_all.RiakHttpTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/fritchie/b/src/riak-python-client/riak/tests/test_all.py", line
839, in test_solr_search_with_params_from_bucket
    self.assertEquals(1, len(list(result.iter("doc"))))
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 250 tests in 27.837s

FAILED (failures=4)

Full test output: https://gist.github.com/2229556

TestServer hangs

I start test server with something like this http://paste.pocoo.org/show/501697/ and it hangs. If you put "print buffer" statement into test_server.py (wait_for_erlang_prompt func) you see something like this: http://paste.pocoo.org/show/501713/

test_delete/store/get_file_with_luwak fail

I installed riaksearch using the latest rpm, and riak-python-client as well.
When I ran python setup.py test, I got the following errors. I'm more than happy to provide other information for debugging if you need.

ERROR: test_delete_file_with_luwak (riak.tests.test_all.RiakHttpTransportTestCase)

Traceback (most recent call last):
File "/data/yard/riak-py/riak/tests/test_all.py", line 816, in test_delete_file_with_luwak
self.client.store_file(key, data)
File "/data/yard/riak-py/riak/client.py", line 299, in store_file
self._transport.store_file(filename, content_type=content_type, content=data)
File "/data/yard/riak-py/riak/transports/http.py", line 347, in store_file
return self.do_put(host, port, url, headers, content, key=key)
File "/data/yard/riak-py/riak/transports/http.py", line 134, in do_put
self.check_http_code(response, [204])
File "/data/yard/riak-py/riak/transports/http.py", line 228, in check_http_code
raise Exception(m)
Exception: Expected status [204], received 404

ERROR: test_store_file_with_luwak (riak.tests.test_all.RiakHttpTransportTestCase)

Traceback (most recent call last):
File "/data/yard/riak-py/riak/tests/test_all.py", line 791, in test_store_file_with_luwak
self.client.store_file(key, data)
File "/data/yard/riak-py/riak/client.py", line 299, in store_file
self._transport.store_file(filename, content_type=content_type, content=data)
File "/data/yard/riak-py/riak/transports/http.py", line 347, in store_file
return self.do_put(host, port, url, headers, content, key=key)
File "/data/yard/riak-py/riak/transports/http.py", line 134, in do_put
self.check_http_code(response, [204])
File "/data/yard/riak-py/riak/transports/http.py", line 228, in check_http_code
raise Exception(m)
Exception: Expected status [204], received 404

ERROR: test_store_get_file_with_luwak (riak.tests.test_all.RiakHttpTransportTestCase)

Traceback (most recent call last):
File "/data/yard/riak-py/riak/tests/test_all.py", line 802, in test_store_get_file_with_luwak
self.client.store_file(key, data)
File "/data/yard/riak-py/riak/client.py", line 299, in store_file
self._transport.store_file(filename, content_type=content_type, content=data)
File "/data/yard/riak-py/riak/transports/http.py", line 347, in store_file
return self.do_put(host, port, url, headers, content, key=key)
File "/data/yard/riak-py/riak/transports/http.py", line 134, in do_put
self.check_http_code(response, [204])
File "/data/yard/riak-py/riak/transports/http.py", line 228, in check_http_code
raise Exception(m)
Exception: Expected status [204], received 404

Enable user's to use their own RiakObject subclass

I propose a bacwards-compatible change to the API that would enable people to supply their own RiakObject subclass to be returned from Riak. Example would be:

class Director(Person):
    ...

class Movie(RiakObject):
    def get_director(self):
        for l in self.get_links():
            if l.get_tag() == 'directed_by':
                return l.get(result_class=Director)
        return None

pf = riak_client.bucket('imdb').get('pulp_fiction', result_class=Movie)
pf.get_director()

I will be happy to provide the patch if you are interested.

search using secondary index fails if key_start == key_end

Just testing 2i so I tryed:

import riak

client = riak.RiakClient(port=8071, transport_class=riak.RiakPbcTransport)

user_bucket = client.bucket('users')
admin_user = user_bucket.new('admin', data={'username': 'admin', 'password': '123'})
admin_user.add_index('username_bin', 'admin')
admin_user.store()

for obj_link in client.index('users', 'username_bin', 'admin').run():
    obj = obj_link.get()
    print obj.get_data()

# WHY THIS SEARCH CAUSES ERROR?

for obj_link in client.index('users', 'username_bin', 'admin', 'admin').run():
    obj = obj_link.get()
    print obj.get_data()

Then I get:

carlo@carlo-laptop:/tmp$ python2.7 testriak.py 
{u'username': u'admin', u'password': u'123'}
Traceback (most recent call last):
  File "testriak.py", line 16, in <module>
    for obj_link in client.index('users', 'username_bin', 'admin', 'admin').run():
  File "/usr/local/lib/python2.7/site-packages/riak/mapreduce.py", line 230, in run
    result = t.mapred(self._inputs, query, timeout)
  File "/usr/local/lib/python2.7/site-packages/riak/transports/pbc.py", line 360, in mapred
    _handle_response)
  File "/usr/local/lib/python2.7/site-packages/riak/transports/pbc.py", line 393, in send_msg_multi
    msg_code, resp = self.recv_msg(conn, expect)
  File "/usr/local/lib/python2.7/site-packages/riak/transports/pbc.py", line 413, in recv_msg
    self.recv_pkt(conn)
  File "/usr/local/lib/python2.7/site-packages/riak/transports/pbc.py", line 460, in recv_pkt
    len(nmsglen))
riak.RiakError: 'Socket returned short packet length 0 - expected 4'

Problems storing objects over PB transport if the message is long.

We see this error when trying to save a Riak object via the PB transport when the object is "big" (i.e. contains an HTML email.)

'PB socket returned short write 62264 - expected 78901'

This happens quite regularly.

The code that produces this is here: https://github.com/basho/riak-python-client/blob/master/riak/transports/pbc.py#L367

Our Riak version is 1.0.1 and our riak-python-client is 1.3.0

occur "KeyError: 'text/html'" when search

I just finish install riak, and riak python client, then I try this:

import riak
client = riak.RiakClient()
test = client.bucket('tests')

test.new('abc', {'a':123}).store()
client.solr().search('tests', {'a':123})

error happen:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.6/dist-packages/riak-1.4.1-py2.6.egg/riak/search.py", line 85, in search
decoder = self.get_decoder(headers['content-type'])
File "/usr/local/lib/python2.6/dist-packages/riak-1.4.1-py2.6.egg/riak/search.py", line 27, in get_decoder
decoder = self._client.get_decoder(content_type) or self._decoders[content_type]
KeyError: 'text/html'

I don't know why~ please!

Please provide a set_links method

It would be really nice to be able to replace all links in bulk. A set_links method that lets you provide a full set of links to replace the current set would be ideal.

No support for streaming results from MapReduce

$TITLE says it all, really. There doesn't seem to be a way to stream results from map phases using this library.

Installation requiring bitbucket login

If I do a pip install riak I am prompted for a bitbucket login:

pip install riak
Downloading/unpacking riak
User for bitbucket.org: 
Password: 
Operation cancelled by user
Storing complete log in /Users/rleland/.pip/pip.log

I started to dig in to see what's going on here. It doesn't appear to be a problem with the riak packaging. From what I've been able to discern from my pip logs there is an issue with the data that is currently on PyPI:

cat ~/.pip/pip.log
------------------------------------------------------------
/Users/rleland/.virtualenvs/riak-testing/bin/pip run on Thu Mar  8 14:47:44 2012
Downloading/unpacking riak==1.3.0
  Getting page http://pypi.python.org/simple/riak
  URLs to search for versions for riak==1.3.0:
  * http://pypi.python.org/simple/riak/1.3.0
  * http://pypi.python.org/simple/riak/
  Getting page http://pypi.python.org/simple/riak/1.3.0
  Getting page https://bitbucket.org/basho/riak-python-client
  Could not fetch URL http://pypi.python.org/simple/riak/1.3.0: HTTP Error 404: Not Found (riak/1.3.0)
  Will skip URL http://pypi.python.org/simple/riak/1.3.0 when looking for download links for riak==1.3.0
  Getting page https://github.com/basho/riak-python-client
  Could not fetch URL https://bitbucket.org/basho/riak-python-client (from http://pypi.python.org/simple/riak/): HTTP Error 401: UNAUTHORIZED
  Will skip URL https://bitbucket.org/basho/riak-python-client when looking for download links for riak==1.3.0
Operation cancelled by user

Looking at that output it just appears that something is goofed on the PyPI end of things. I'm wondering if just re-running python setup.py sdist register upload will fix it up. Obviously I can't do that but let me know when you do and I'll test from my end.

Add support for setting timeouts on sockets

Curently it's not possible to set timeouts for sockets Riak client uses for PB connections. This results in clients that hang for extended periods if a server disappears and nothing between the client and the server closes the TCP connection.

Adding timeouts is somewhat problematic API-wise, since there is no way to pass extra options to the connection class currently.

Key not set on objects whose key is generated by Riak

The HTTP transport offers the ability to let Riak generate a random key. When it does, the Location header reveals the randomly generated key. It would be really nice if this was parsed and the key was set on the RiakObject object.

riak.RiakError: 'Socket returned short packet 135 - expected 1213486160' error back?

I've followed the Fast Track and am trying to run the example script (http://basho.github.com/riak-python-client/tutorial.html#quick-start) with the following modification:

# Connect to Riak.
client = riak.RiakClient('127.0.0.1', port=8091, transport_class=riak.RiakPbcTransport)

and I'm getting:

Traceback (most recent call last):
  File "riak_test.py", line 18, in <module>
    person.store()
  File "build/bdist.macosx-10.6-universal/egg/riak/riak_object.py", line 269, in store
  File "build/bdist.macosx-10.6-universal/egg/riak/transports/pbc.py", line 190, in put
  File "build/bdist.macosx-10.6-universal/egg/riak/transports/pbc.py", line 349, in recv_msg
  File "build/bdist.macosx-10.6-universal/egg/riak/transports/pbc.py", line 401, in recv_pkt
riak.RiakError: 'Socket returned short packet 135 - expected 1213486160'

I've found a reference for this (https://issues.basho.com/show_bug.cgi?id=695) but I'm running d3c25d9 which includes the fix, doesn't it?

Any advice appreciated,
Sven

bucket.enable_seach() doesn't work with the PBC transport

If I enable search via the python client over the HTTP interface everything works fine but if I enable it over the PBC transport there doesn't seem to be any effect. See the gists below for sample runs that illustrate the problem.

https://gist.github.com/1151661

Note: In the gist I use the test server provided by the client but I've also replicated the same bug with a normal instance of the latest "stable" release of riak-search.

choose a consistent style

We have some code with 4 space indentation, some with 2. We should agree on something going forward. Maybe PEP 8?

Allow user to set R/W/etc when instantiating Client, Bucket, etc.

This shouldn't be a hard change, I can probably add it quickly.

First, I'm just curious if this makes sense. It seems obvious to me, but maybe I'm missing a reason why we shouldn't do this?

Protobuf transport doesn't recover gracefully from servers going up and down

A client using the Protobuf transport class is permanently wedged if it ever has connection issues. It would be great if the connection were re-created or cleaned up after any error (a fresh attempt on each command after a connection failure?).

In addition, it seems to me that an is_alive() failure should return False and not an exception. Thoughts?

>>> # riak isn't running

>>> import riak

>>> client = riak.RiakClient(port=8087, transport_class=riak.RiakPbcTransport)

>>> client.is_alive()
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)

/home/brett/Development/python/disqus.com/git/disqus/disqus/<ipython console> in <module>()

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/client.py in is_alive(self)
    237         :rtype: boolean
    238         """
--> 239         return self._transport.ping()
    240 
    241     def add(self, *args):

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in ping(self)
    103         @return boolean
    104         """
--> 105         self.maybe_connect()
    106         self.send_msg_code(MSG_CODE_PING_REQ)
    107         msg_code, msg = self.recv_msg()

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in maybe_connect(self)
    333         if self._sock is None:
    334             self._sock = s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
--> 335             s.connect((self._host, self._port))
    336             if self._client_id:
    337                 self.set_client_id(self._client_id)

/usr/lib/python2.6/socket.pyc in connect(self, *args)

error: [Errno 111] Connection refused

>>> # start riak

>>> client.is_alive()
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)

/home/brett/Development/python/disqus.com/git/disqus/disqus/<ipython console> in <module>()

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/client.py in is_alive(self)
    237         :rtype: boolean
    238         """
--> 239         return self._transport.ping()
    240 
    241     def add(self, *args):

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in ping(self)
    104         """
    105         self.maybe_connect()
--> 106         self.send_msg_code(MSG_CODE_PING_REQ)
    107         msg_code, msg = self.recv_msg()
    108         if msg_code == MSG_CODE_PING_RESP:

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in send_msg_code(self, msg_code)
    339     def send_msg_code(self, msg_code):
    340         pkt = struct.pack("!iB", 1, msg_code)
--> 341         self._sock.send(pkt)
    342 
    343     def encode_msg(self, msg_code, msg):

error: [Errno 32] Broken pipe

Here is a slightly different example,

>>> # riak is running

>>> import riak

>>> client = riak.RiakClient(port=8087, transport_class=riak.RiakPbcTransport)

>>> client.is_alive()
1

>>> # stop riak

>>> client.is_alive()
---------------------------------------------------------------------------
RiakError                                 Traceback (most recent call last)

/home/brett/Development/python/disqus.com/git/disqus/disqus/<ipython console> in <module>()

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/client.py in is_alive(self)
    237         :rtype: boolean
    238         """
--> 239         return self._transport.ping()
    240 
    241     def add(self, *args):

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in ping(self)
    105         self.maybe_connect()
    106         self.send_msg_code(MSG_CODE_PING_REQ)
--> 107         msg_code, msg = self.recv_msg()
    108         if msg_code == MSG_CODE_PING_RESP:
    109             return 1

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in recv_msg(self)
    355 
    356     def recv_msg(self):
--> 357         self.recv_pkt()
    358         msg_code, = struct.unpack("B", self._inbuf[:1])
    359         if msg_code == MSG_CODE_ERROR_RESP:

/home/brett/Development/python/disqus.com/lib/python2.6/site-packages/riak-1.2.2_disqus1-py2.6.egg/riak/transports/pbc.py in recv_pkt(self)
    399         if len(nmsglen) != 4:
    400             raise RiakError("Socket returned short packet length {0} - expected 4".
--> 401                             format(nmsglen))
    402         msglen, = struct.unpack('!i', nmsglen)
    403         self._inbuf_len = msglen

RiakError: 'Socket returned short packet length  - expected 4'

Support the head parameter if pbc transport is used. [JIRA: CLIENTS-1039]

Rationale: If the key management is out sourced to other systems and huge deletes should be performed there is no need for any data to be transfered. The implementation of this parameter would greatly improve this sort of task by reducing used resources and a noticeable speed up - depending on the data size.

By the way, why is this parameter only available for the pbc api ?

Beside this implementation suggestion, I would generally love to see a much lighter interface for operations which depends only on the key. Specifically I talk about something like:

bucket.delete(key)
bucket.exists(key)

and would just return boolean as no other informations are needed.

test_store_of_missing_object failed for riaksearch

Hi,
I installed riak-search-0.14.0-1.el5.x86_64.rpm, and sync the riak-python-client from git.
When I ran python setup.py test, all but 1 test failed:
====================================================================== [0/15622]

ERROR: test_store_of_missing_object (riak.tests.test_all.RiakPbcTransportTestCase)

Traceback (most recent call last):
File "/data/users/byzhang/software/riak-py/riak/tests/test_all.py", line 413, in test_store_of_missing_object
o = o.store()
File "/data/users/byzhang/software/riak-py/riak/riak_object.py", line 296, in store
Result = t.put(self, w, dw, return_body)
File "/data/users/byzhang/software/riak-py/riak/transports/pbc.py", line 190, in put
msg_code, resp = self.recv_msg()
File "/data/users/byzhang/software/riak-py/riak/transports/pbc.py", line 360, in recv_msg
self.recv_pkt()
File "/data/users/byzhang/software/riak-py/riak/transports/pbc.py", line 404, in recv_pkt
format(nmsglen))
RiakError: 'Socket returned short packet length - expected 4'

---

I installed protobuf from the latest source, with PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp.
But I saw the same error after clear that env.

I ran the commands inside test_store_of_missing_object in python console, but using httptransport. They works fine.

I also tried to install the newer version of riak-search-0.14.2-1.el5.x86_64.rpm, however, it seems all tests failed (due to Connection refused, and the riaksearch is really crashed) after this one:

ERROR: test_generate_key (riak.tests.test_all.RiakHttpTransportTestCase)

Traceback (most recent call last):
File "/data/users/byzhang/software/riak-py/riak/tests/test_all.py", line 717, in test_generate_key
bucket.new(None, data={}).store()
File "/data/users/byzhang/software/riak-py/riak/riak_object.py", line 296, in store
Result = t.put(self, w, dw, return_body)
File "/data/users/byzhang/software/riak-py/riak/transports/http.py", line 122, in put
response = self.http_request('POST', host, port, url, headers, content)
File "/data/users/byzhang/software/riak-py/riak/transports/http.py", line 366, in http_request
return self.httplib_request(method, host, port, url, headers, obj)
File "/data/users/byzhang/software/riak-py/riak/transports/http.py", line 377, in httplib_request
response = client.getresponse()
File "/var/tmp/Python-1-fb-buildroot/usr/local/lib/python2.6/httplib.py", line 950, in getresponse
response.begin()
File "/var/tmp/Python-1-fb-buildroot/usr/local/lib/python2.6/httplib.py", line 390, in begin
version, status, reason = self._read_status()
File "/var/tmp/Python-1-fb-buildroot/usr/local/lib/python2.6/httplib.py", line 354, in _read_status
raise BadStatusLine(line)
BadStatusLine

Can you kindly give some suggestion on these errors?

Thanks,
-B

Port number given as String is not catched

If you set a string as port number for the Riak connection it sometimes failes with a strange error like:

gaierror: [Errno -8] Servname not supported for ai_socktype

Broken via inspection

I have no idea how this can work, given that I see no client_id anywhere in scope.
https://github.com/basho/riak-python-client/blob/master/riak/search.py#L13

store() fails if object does not exist

If I try to update a key that did not exist, I get an error when trying to store that object:

>>> import riak
>>> c = riak.RiakClient(port=8091)
>>> b = c.bucket("test")
>>> o = b.get("foo")
>>> o.exists()
False
>>> o.set_data({"foo": "bar"})
<riak.riak_object.RiakObject object at 0x100676950>
>>> o.store()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/emoritz/Projects/riak/riak-python-client/riak/riak_object.py", line 269, in     store
    Result = t.put(self, w, dw, return_body)
  File "/Users/emoritz/Projects/riak/riak-python-client/riak/transports/http.py", line 100, in put
    'Content-Type' : robj.get_content_type(),
  File "/Users/emoritz/Projects/riak/riak-python-client/riak/riak_object.py", line 172, in get_content_type
    return self._metadata[MD_CTYPE]
KeyError: 'content-type'

Split unit test

Hi, I saw that the unit tests are very aggregate (one depend on the other).

you think is important to refactor the test to make them independent?

PB connections don't have timeouts

PB connections do not have timeouts. This causes the client to hang indefinitely when the server it's connected to disappears.

TestServer waits forever if it fails to start

If a TestServer instance fails to start for any reason, it waits for an Erlang console prompt which never arrives.

setup.py tests doesn't work and tests don't pass

I wanted to start some refactoring but that'd be silly without ensuring tests currently pass, right? :) Going by the docs...

python setup.py tests
AttributeError: 'TestLoader' object has no attribute 'discover'

https://github.com/basho/riak-python-client/blob/master/riak/tests/suite.py#L9

Are you depending on a monkey patch or something I don't have installed?

If I give up and try running tests/test_all.py myself I get 17 failures (I won't paste them here until I'm sure this is how I should be running them).

remove protobuffer dependency from setup.py

If you try to install it with pip you get an error:

IOError: [Errno 2] No such file or directory: '/Users/patricklauber/Documents/workspace/LogDeamon/build/protobuf/setup.py'

Pelase remove this 2 line from the setup.py... There should be only python programs in there.

install_requires = ['protobuf>=2.3.0'],
dependency_links = ["http://downloads.basho.com/support"],

test_siblings test fails

I believe this has to do with Riak 1.0/vnode_vclocks, but test_siblings fails pretty consistently.

======================================================================
ERROR: test_siblings (riak.tests.test_all.RiakPbcTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sean/Development/riak-python-client/riak/tests/test_all.py", line 254, in test_siblings
    obj = bucket.get('foo')
  File "/Users/sean/Development/riak-python-client/riak/bucket.py", line 316, in get
    return obj.reload(r=r, pr=pr)
  File "/Users/sean/Development/riak-python-client/riak/riak_object.py", line 445, in reload
    self.populate(Result)
  File "/Users/sean/Development/riak-python-client/riak/riak_object.py", line 531, in populate
    self.set_encoded_data(data)
  File "/Users/sean/Development/riak-python-client/riak/riak_object.py", line 139, in set_encoded_data
    self._data = decoder(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 338, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

======================================================================
FAIL: test_siblings (riak.tests.test_all.RiakHttpTransportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sean/Development/riak-python-client/riak/tests/test_all.py", line 278, in test_siblings
    self.assertEqual(obj.get_sibling_count(), 5)
AssertionError: 6 != 5

Cannot install 1.5.0 using pip

Let me preface this with I might be a moron and missing something obvious.

On a clean Ubuntu Lucid install and on a Mountain Lion install of OSX, pip install riak yields v1.4.1. pip search riak shows 1.5.0 as LATEST, but pip install riak==1.5.0 blows up saying it cannot find the package.

The pip logs indicate that pip is looking at http://pypi.python.org/simple/riak and parsing the links looking for a riak-1.5.0.tar.gz.

Not entirely sure what's going on here though. Anyone else seeing this?

Request: riak.mapreduce.RiakMapReduce.add() to accept list of bucket-key pairs

Currently, it looks like riak.mapreduce.RiakMapReduce.add() will accept a few variations, including a bucket and a bucket-key pair, but not a list of buckey-key pairs (similar to a JSON POST to /mapred). This would be a convenient feature.

Riak python client will fail to insert data if Key is not provided

Running the code below will fail with an exception. Riak Python library should
provide a method to store an object without key provided.

Code:

import riak
client = riak.RiakClient()

bucket = client.bucket('test')

person = bucket.new(None, data={
'name': 'John Smith',
'age': 28,
'company': 'Mr. Startup!',
})

Save the object to Riak.

person.store()

Exception:

File "learn_riak.py", line 18, in
person.store()
File "build/bdist.linux-i686/egg/riak/riak_object.py", line 269, in store
File "build/bdist.linux-i686/egg/riak/transports/http.py", line 118, in put
File "build/bdist.linux-i686/egg/riak/transports/http.py", line 341, in
http_request
File "build/bdist.linux-i686/egg/riak/transports/http.py", line 351, in
httplib_request
File "/usr/lib/python2.6/httplib.py", line 910, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.6/httplib.py", line 947, in _send_request
self.endheaders()
File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 735, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 716, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 514, in create_connection
raise error, msg
socket.error: [Errno 111] Connection refused

Error handling in python client [JIRA: CLIENTS-667]

Moved from https://issues.basho.com/show_bug.cgi?id=385, reported by Mehmet Ali Akmanalp

It looks like it's hard to handle or even notice errors with the python client. Here's a prime example:

import riak

c = riak.RiakClient("127.0.0.1", 8087, transport_class=riak.RiakPbcTransport)

b = c.bucket("")
o = b.new("")
o.set_data({"foo":1, "bar": 4})
o.store()

This runs silently. No exceptions are thrown. I don't know what it did. I can't check any of the return values because they mostly return "self". There seems to be a RiakError but that's a more generic "oh crap something went terribly wrong" whereas the sort of granularity people would want (IMHO) is "the database connection failed" (which throws a socket.error by the way) or "x does not exist".

Thanks,
~mali

unittest fails with AttributeError: 'FileDescriptor' object has no attribute 'message_types_by_name'

Everytime I try to run python setup.py test

Traceback (most recent call last):
  File "setup.py", line 46, in <module>
    'Topic :: Database']
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.24-py2.7.egg/setuptools/command/test.py", line 137, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.24-py2.7.egg/setuptools/command/test.py", line 117, in with_project_on_sys_path
    func()
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.24-py2.7.egg/setuptools/command/test.py", line 146, in run_tests
    testLoader = loader_class()
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 128, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "/home/shuhao/Dropbox/RandomProjects/riak-python-client/riak/__init__.py", line 41, in <module>
    from client import RiakClient
  File "/home/shuhao/Dropbox/RandomProjects/riak-python-client/riak/client.py", line 28, in <module>
    from riak.search import RiakSearch
  File "/home/shuhao/Dropbox/RandomProjects/riak-python-client/riak/search.py", line 1, in <module>
    from riak.transports import RiakHttpTransport
  File "/home/shuhao/Dropbox/RandomProjects/riak-python-client/riak/transports/__init__.py", line 2, in <module>
    from pbc import RiakPbcTransport
  File "/home/shuhao/Dropbox/RandomProjects/riak-python-client/riak/transports/pbc.py", line 50, in <module>
    import riakclient_pb2
  File "/home/shuhao/Dropbox/RandomProjects/riak-python-client/riak/transports/riakclient_pb2.py", line 964, in <module>
    DESCRIPTOR.message_types_by_name['RpbErrorResp'] = _RPBERRORRESP
AttributeError: 'FileDescriptor' object has no attribute 'message_types_by_name'

Python 3 Support [JIRA: CLIENTS-50]

Couldn't see anywhere better to do this, so I'll do it here.
Any plans for Python 3 support soon?

riakobj.get_data returns dictionary reference

riakobj.get_data returns the reference to the internal data dictionary. It should really return a deepcopy of it or at least a shallow copy of it.