keboola / sapi-python-client Goto Github PK
View Code? Open in Web Editor NEWKeboola Connection Storage API client
Home Page: http://docs.keboola.apiary.io/
License: MIT License
Keboola Connection Storage API client
Home Page: http://docs.keboola.apiary.io/
License: MIT License
Currently application/x-www-form-urlencoded
is used everywhere. While acceptable, there may be a problem in requests, which are potentially large (such as saving/upadting configurations). There multipart/form-data
has to be used. A simple solution is to use multipart/form-data
everywhere disregarding a slight performance hit. This is probably important only for POST
requests.
should be installable as a package
client.tables.export_to_file('in.c-test.sample_table', 'out/tables', is_gzip=True)
results in something like this:
"new_opt_account_vwocategorypicklist","ema_IsfromSuite"
?ٿ�&��$�������K�����uR�IQ�?�dB�e4>x�Ң��x�|>����2YΆTN�|�<'@��9����Y������^X�� jv
ungzipping the output table without header works fine. What are the options here?
Hello guys,
I believe there is a minor bug in the computation of sleep time between successive API calls.
The retries variable does not get incremented and therefore the sleep time is always 2 seconds.
This does not impact me in any significant way. I just noticed that this might not have been the original intention and wanted to let you know.
sapi-python-client/kbcstorage/jobs.py
Lines 100 to 121 in 5a93926
I am trying to import data to keboola from csv with load
function in class Table
. According to documentation, it should create table, if not exist:
def load(self, table_id, file_path, is_incremental=False, delimiter=',',
enclosure='"', escaped_by='', columns=None,
without_headers=False):
"""
Create a new table from CSV file.
When I trying to import data to non-exists table, I receive an error 404 Client Error: Not Found for url
. Is this normal and load function should be only for update existing table ? Now, I have to check if table exist, if not, create it and than load the data. According to me, it should create the table automatically.
Hi guys, seems awsome and I'd love to contribute my few cents.
Is there a recommended workflow for contributions? I'd like to ask b4 I start working on a feature.
Right now I have some refactoring ideas. Would you prefer discussing them in advance or should I just make the changes and discuss them in a PR?
Mainly I would
camelCase
to pep8_case
Client
class a sublcass of the HttpHelper
to avoid the awkard calls such as client.http.get_something()
(The cleaner version would be client.get_something()
.HttpHelper.tokenheader()
into a @property
. Imo it makes more sense to use it like HttpHelper.tokenheader
This would change the api alot and break lot of existing code so I'm asking in advance if you are ok with that.
That's what I got just through quickly skimming the source, feel free to disagree with anything 🍺
This thing needs tests.
class Client:
def __init__(self, token, api_domain='https://connection.keboola.com'):
pass
so that you can init the client like this
client = Client('your-token')
What are the alternative api_domains anyway?
======================================================================
FAIL: test_create_file (tests.functional.test_files.TestFiles) ()
----------------------------------------------------------------------
Traceback (most recent call last):
File "/code/tests/functional/test_files.py", line 67, in test_create_file
self.assertFalse('credentials' in file_info)
AssertionError: True is not false
when passing, remove https://github.com/keboola/sapi-python-client/blob/master/.travis.yml#L8
to allow uploading larger files
https://github.com/keboola/storage-api-php-client/blob/master/src/Keboola/StorageApi/Client.php#L1310
Hi,
I believe I have spotted a bug in the export_to_file() method of the Tables class.
The exported csv always gets headers based on all columns of the table even if I select only a subset of them. Moreover, the order of header labels is based purely on the table detail in KBC, even though they might be exported in different order based on the columns argument.
I think this could be fixed if you used columns from table detail only as a fallback to handle situations where columns are not specified by the user. I.e. by replacing line 422 in kbcstorage/tables.py by this:
if columns is None:
columns = table_detail['columns']
doesn't pass with the latest flake
You may have noticed I've looking at the tests ( #43 #44 #45 #46) - Hope these prs are welcome! I want to make a few more enhancements to the tests to make them slightly easier to work with, and then I'll look working on the issues in the milestone for v1.0.0.
as discussed in some comments floating somewhere around here. The client would handle passing of the base_url
and auth_token
to the endpoints + bundle all the endpoints to one place so we are left with this simple and clean usage:
>>> from kbcstorage import Client
>>> client = Client('myapitoken')
>>> client = Client('myapitoken', root_url='https://connection.keboola.com') # alternatively
>>> client.buckets.list()
# a list of all buckets
>>> client.workspaces.detail(1234)
# etc...
This still allows one to instantiate endpoints one by one if he wishes to.
from kbcstorage.tables import Tables
from kbcstorage.buckets import Buckets
tables = Tables('https://connection.keboola.com', 'your-token')
# get table data into local file
tables.export_to_file(table_id='in.c-demo.some-table', path_name='/data/')
# save data
tables.create(name='some-table-2', bucket_id='in.c-demo', file_path='/data/some-table')
# list buckets
buckets = Buckets('https://connection.keboola.com', 'your-token')
buckets.list()
# list bucket tables
buckets.list_tables('in.c-demo')
# get table info
tables.detail('in.c-demo')
I am happy to implement this after we deal with #28
Let's discuss (feel free to stop me if there was too much discussion and no code!) the client architecture here
As mentioned in #3 I'd suggest structuring out the client in the following spirit (this is just a mock-up):
class Endpoint:
"""Endpoint implements all HTTP methods
serves as a base class for the real endpoints.
"""
def __init__(self, root, path, token):
self.path = path
def _get(**params):
return requests.get(
urllib.parse.urljoin(root, path),
headers={'x-storage-token':self.token},
params=params)
# either make a subclass
class BucketEndpoint(Endpoint):
def listBuckets(self, *args, **kwargs):
return self._get(*args, **kwargs)
class StorageClient:
def __init__(self, token):
self.token = token
self.root = 'http://something.com'
self.buckets = BucketsEndpoint(self.root, 'buckets'):
# and use like this:
client = StorageClient('apitoken')
client.buckets.listBuckets(id='1234')
but @Ogaday mentioned he was developing his own client, so he might already have an alternative. How do you feel about it?
sapi-python-client/kbcstorage/tables.py
Line 509 in 276bceb
limit should be int but validation checks table id instead
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.