okfn / ckanclient-deprecated

DEPRECATED - please see https://github.com/ckan/ckanapi. [Python client library for CKAN]

ckanclient

ckanclient is a Python library and command-line client to read and write to a CKAN instance via the API. It supports the CKAN API, the CKAN FileStore's Storage API, and the CKAN DataStore's Data API.

Installation

Using pip:

pip install ckanclient

You can also download releases of ckanclient directly from PyPI:

http://pypi.python.org/pypi/ckanclient

or you can get the latest development version from git:

git clone https://github.com/okfn/ckanclient.git

Usage

API Key

You can either pass your API key explicitly when creating the CkanClient instance:

client = CkanClient(api_key='my-api-key')

Or you can put your API key in your ~/.ckanclientrc file:

[index:{hostname}]
api_key = {your-api-key}

For example:

[index:datahub.io]
api_key = adfakjfafdkjda

[index:localhost]
api_key = tester

Python Library

Catalog API

ckanclient can be used to make requests to the CKAN API, including the API's REST interface to all primary objects (datasets, groups, tags) and its search interface.

The simplest way to make CKAN API requests is:

import ckanclient

# Instantiate the CKAN client.
ckan = ckanclient.CkanClient(base_location='http://datahub.io/api',
                             api_key='adbc-1c0d')
# (use your own api_key from http://datahub.io/user/me )

# Get the package list.
package_list = ckan.package_register_get()
print package_list

# Get the tag list.
tag_list = ckan.tag_register_get()
print tag_list

# Collect the package metadata.
package_entity = {
    'name': my_package_name,
    'url': my_package_url,
    'download_url': my_package_download_url,
    'tags': my_package_keywords,
    'notes': my_package_long_description,
}

# Register the package.
ckan.package_register_post(package_entity)

# Get the details of a package.
ckan.package_entity_get(package_name)
package_entity = ckan.last_message
print package_entity

# Update the details of a package.
ckan.package_entity_get(package_name)
package_entity = ckan.last_message
package_entity['url'] = new_package_url
package_entity['notes'] = new_package_notes
ckan.package_entity_put(package_entity)

# List groups
group_list = ckan.group_register_get()
print group_list

# Create a new group
group_entity = {
    'name': my_group_name,
    'title': my_group_title,
    'description': my_group_description,
    'packages': group_package_names,
    }
ckan.group_register_post(group_entity)

# Get the details of a group.
print ckan.group_entity_get(group_name)

# Update the group details
group_entity = ckan.last_message
group_entity['title'] = new_group_title
group_entity['packages'] = new_group_packages
ckan.group_entity_put(group_entity)

FileStore and Storage API

You can use ckanclient to upload files to CKAN's FileStore via the Storage API, example:

import ckanclient
ckan = ckanclient.CkanClient(base_location='http://datahub.io/api',
    api_key='aa9368b2-6f18-4c96-b190-4f3355613d88')
ckan.upload_file('my_data.csv')
ckan.add_package_resource('my_dataset', 'my_data_file.csv',
            resource_type='data', description='...')
ckan.add_package_resource('my_dataset', 'http://example.org/foo.txt',
            name='Foo', resource_type='metadata', format='csv')

DataStore and Data API

To be updated - read the source for the present!

Command Line Interface

Install ckanclient will create a command line client named (unsurprisingly!) ckanclient. To see usage do:

ckanclient -h

Example:

ckanclient package_entity_get ckan

You can specify the ckan site you wish to use using the --ckan option:

ckanclient --ckan=http://datahub.io/api package_entity_get ckan

NB: the command line is currently under development.

Tests

The ckanclient tests require the ckan and nose modules to be installed. Optionally ckanext-dgu can be installed too and the form api will be tested.

To run the tests:

nosetests --ckan ckanclient/tests

ckanclient-deprecated's Issues

Command line interface

A simple command line interface would be nice!

Python 3.2 support

I am wondering whether support for Python 3.2 is planned already and what help would be wanted for it.

The 2to3 script doesn't quite cut it, as I am left in a state where urllib2 is still referenced (which is merged into urllib in Python 3) and after importing the proper libraries (changing urllib2 to urllib.request in ckanclient/init.py) there remains some errors with getting the registry from an api.

I haven't tried any further than that currently.

I'd be happy to help out in migrating but would just like to check whether someone else is on this task already.

problems with ~/.ckanclientrc

Greetings,

I'm having some trouble getting ~/.ckanclientrc to work as https://github.com/okfn/ckanclient suggests.

While the following code works fine, it does not work when I remove the api_key argument -- even if I have specified the key in ~/.ckanclientrc (shown below).

How can I get the CkanClient to use the API key given in ~/.ckanclientrc?

Thanks for much!
Tim Lebo

import ckanclient

# Instantiate the CKAN client.
ckan = ckanclient.CkanClient(base_location='http://datahub.io/api',
                            api_key='my-key') ## 403 error if this argument is removed (why isn't ~/.ckanclientrc taking over?).

# Get the details of a package.
ckan.package_entity_get('instance-hub-fiscal-years')
package_entity = ckan.last_message
package_entity['maintainer'] = 'TWC RPI'

print package_entity

ckan.package_entity_put(package_entity)

cat ~/.ckanclientrc
[index:http://datahub.io]
api_key = my-key

Update ckanclient for new DataStore

Rename package -> dataset everywhere

Redirected to another URL error when trying to upload file using ckanclient

0 down vote favorite

I am trying to upload a json file to my acount in Datahub.io with ckanclient, and I got this error:

ckanclient.CkanApiError: Got redirected to another URL, which does not work with POSTS. Redirection: http://thedatahub.org/api/storage/auth/form/2013-07-26T022918/my.json -> http://datahub.io/user/login?came_from=http://datahub.io/api/storage/auth/form/2013-07-26T022918/my.json

The code I use is this (This is the same code as in the API)

import ckanclient

ckan = ckanclient.CkanClient(api_key='8a482c44-ef3d-4337-88c6-dca16fd6b3f3',)
ckan.upload_file('my.json')

What am I doing wrong?

Does anyone have some IPython Notebooks they could share using ckanclient?

Does anyone have some IPython Notebooks they could share using ckanclient? My google search of "ckan ext:ipynb" didn't turn up anything except for a notebook I created to test the CKAN CSW service.

Thanks,
Rich

Refactor to use requests library

http://www.python-requests.org/

Merge in datastore client

https://github.com/okfn/datastore-client

Add complex example of package_search to documentation

It would be great to have a complex example of package_search added to the documentation.

I can get this to work:

q = 'tags:sea_water_temperature'
ckan = ckanclient.CkanClient('http://catalog.data.gov/api/3')
d = ckan.action('package_search', q=q, rows=10)

but can't figure out how to pass a more complex request like

d={"q":"sea_water_temperature",
   "extras":{"ext_bbox":"-121,45,-120,46"},
   "fq": "res_format:WMS"}

Remove Form API support

This is deprecated and no longer supported in CKAN and was not used by anyone but DGU (who have long stopped using it!).

Refactor to have ckanclient.CatalogClient and ckanclient.DataStoreClient

Want to change:

import ckanclient -> import ckanclient.CatalogClient
import datastore -> import ckanclient.DataStoreClient

Move stuff out of ckanclient/__init__.py into catalog.py:

ckanclient/
  catalog.py
  datastore.py

Maybe also do the samething with the FileStore, i.e. have ckanclient.FileStoreClient.

What to do about contrib and loaders? Anything?

Move to ckan organization

@kindly @seanh wdyt?

Also should we not attempt to merge this with ckanapi (or deprecate this in favour of ckanapi?)

update default API URL to datahub.io

https://github.com/okfn/ckanclient/blob/master/ckanclient/__init__.py#L85

is currently base_location = 'http://thedatahub.org/api' but should be base_location = 'http://datahub.io/api'

When POSTing, the old URL's redirection cannot be followed.

Thanks,
Tim

Add and document command-line interfaces for CKAN API and Storage API

File upload borked

It looks like, at least with datahub.io, file upload has not worked since 183ad05 - using this revision is fine.

Any commit since then makes google return an error about posting to create a bucket not being allowed.

Problem when handling documents with unicode

Following ckanclient code(at '_encode_multipart_formdata' in init.py), the code making multipart-form body with document having unicode characters can't process, because Python occurs a error during process converting document using UNICODE into ASCII to make multipart-form body.)

To process document included some unicode characters, it need to modify mulitpart-form processing code that is able to handle UNICODE characters.

ckanclient/init.py

…
import logging
logger = logging.getLogger('ckanclient')


PAGE_SIZE = 10


# from [ 
class MultipartFormdataEncoder(object):
    def __init__(self):
        self.boundary = uuid.uuid4().hex
        self.content_type = 'multipart/form-data; boundary={}'.format(self.boundary)


    @classmethod
    def u(cls, s):
        if sys.hexversion < 0x03000000 and isinstance(s, str):
            s = s.decode('utf-8')
        if sys.hexversion >= 0x03000000 and isinstance(s, bytes):
            s = s.decode('utf-8')
        return s


    def iter(self, fields, files):
        """
        fields is a sequence of (name, value) elements for regular form fields.
        files is a sequence of (name, filename, file-type) elements for data to be uploaded as files
        Yield body's chunk as bytes
        """
        encoder = codecs.getencoder('utf-8')
        for (key, value) in fields:
            key = self.u(key)
            yield encoder('--{}\r\n'.format(self.boundary))
            yield encoder(self.u('Content-Disposition: form-data; name="{}"\r\n').format(key))
            yield encoder('\r\n')
            if isinstance(value, int) or isinstance(value, float):
                value = str(value)
            yield encoder(self.u(value))
            yield encoder('\r\n')
        for (key, filename, fd) in files:
            key = self.u(key)
            filename = self.u(filename)
            yield encoder('--{}\r\n'.format(self.boundary))
            yield encoder(self.u('Content-Disposition: form-data; name="{}"; filename="{}"\r\n').format(key, filename))
            yield encoder('Content-Type: {}\r\n'.format(mimetypes.guess_type(filename)[0] or 'application/octet-stream'))
            yield encoder('\r\n')
            with fd:
                buff = fd.read()
                yield (buff, len(buff))
            yield encoder('\r\n')
        yield encoder('--{}--\r\n'.format(self.boundary))


    def encode(self, fields, files):
        body = io.BytesIO()
        for chunk, chunk_len in self.iter(fields, files):
            body.write(chunk)
        return self.content_type, body.getvalue()
# to ]

class CkanApiError(Exception): pass
…
…
    #
    # Private Helpers
    #
    def _post_multipart(self, url, fields, files):
        '''Post fields and files to an http host as multipart/form-data.


        Taken from
        http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/


        :param fields: a sequence of (name, value) tuples for regular form
            fields
        :param files: a sequence of (name, filename, value) tuples for data to
            be uploaded as files


        :returns: the server's response page


        '''
        content_type, body = MultipartFormdataEncoder().encode(fields, files) # modified this line
        headers = {'Content-Type': content_type}


        # If we got a relative url from api, and we need to build a absolute
        url = urlparse.urljoin(self.base_location, url)


        # If we are posting to ckan, we need to add ckan auth headers.
        if url.startswith(urlparse.urljoin(self.base_location, '/')):
            headers.update({
                'Authorization': self.api_key,
                'X-CKAN-API-Key': self.api_key,
            })


        request = Request(url, data=body, headers=headers)
        response = urlopen(request)
        return response.getcode(), response.read()
…

upload_file method does not support non-ASCII content

The upload_file method uses a faulty snippet of ActiveState code that will choke on any non-ASCII (or, god forbid, binary) content.

I'm using requests.post() as a workaround for now, but it would be nice for this functionality to actually work.

Update file in existing resource.

Uploading a file currently adds a new resource.
Developers need to be able to query the same resource (using the unique resource id), but with up-to-date content within the file.
Call needed for updating/uploading a file within an existing resource.

setup.py should install all dependencies

I was getting error 302 when using CkanClient from a remote box, but not when using from the box where CKAN was installed. After a lot of tests, I've finally got to the point where I copied the full virtualenv from one box to the other.

In a nutshell: setup.py fails to install all needed dependencies.

The error 302 is absolutely misleading. Worse than giving no information is giving wrong/misleading information. If CkanClient tries to import something which is not there or is wrong version, it should fail and state the fact, instead of swallowing the exception, go ahead and fail later due some other reason.

Page count in generator for package_search is incorrect

The page count calculation in package_search is incorrect and should probably just be

page_count = (len(records) + size_of_page - 1) / size_of_page;

Add resource_search method to the CkanClient class

Add a method reflecting CKAN API resource search mechanism.
Example (from CKAN API doc):
http://demo.ckan.org/api/3/action/resource_search?query=name:District%20Names

Do a new release

With the new datastore code in here plus code for filestore we should do a new release.

Release command:

python setup.py register sdist upload