A Python HTTP client for the Kinto API.
License: Other
Kinto is a minimalist JSON storage service with synchronisation and sharing abilities.
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
I'm not sure if supporting plugins is in the scope of the Python client, but I think it would be useful to have attachments support.
My idea is to do this through an argument "files" on create, update and patch record methods.
It would also be good to raise an appropriate exception message if the server doesn't support the plugin.
Today create if not exist is handled that way:
try:
remote_client.create_bucket(
permissions=self.bucket_permissions)
except KintoException as e:
if not hasattr(e, 'response') or e.response.status_code != 412:
raise e
It could be handled like that:
remote_client.create_bucket(permissions=self.bucket_permissions, if_not_exists=True)
When bucket and collection are specified in client
Update record with id 'fdf007bf-10ea-4e55-b3eb-eebd12c87c88' in collection None in bucket None
Different things to do:
>>> cli = Client(server_url='https://kinto.dev.mozaws.net/v1/', auth=('gabi', 'my_secret'))
>>> cli
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/kinto_http/__init__.py", line 53, in get
return pattern.format(root=self._root, **kwargs)
KeyError: 'collection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/kinto_http/__init__.py", line 529, in __repr__
collection=self._collection_name
File "/usr/local/lib/python3.6/site-packages/kinto_http/__init__.py", line 120, in get_endpoint
return self.endpoints.get(name, **kwargs)
File "/usr/local/lib/python3.6/site-packages/kinto_http/__init__.py", line 57, in get
field=','.join(e.args)))
kinto_http.exceptions.KintoException: (KintoException(...), 'Cannot get collection endpoint, collection is missing')$ make install
virtualenv .venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in .venv/bin/python2
Also creating executable in .venv/bin/python
Installing setuptools, pip...done.
.venv/bin/python setup.py develop
running develop
running egg_info
creating kinto_client.egg-info
writing requirements to kinto_client.egg-info/requires.txt
writing kinto_client.egg-info/PKG-INFO
writing top-level names to kinto_client.egg-info/top_level.txt
writing dependency_links to kinto_client.egg-info/dependency_links.txt
writing manifest file 'kinto_client.egg-info/SOURCES.txt'
reading manifest file 'kinto_client.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'kinto_client.egg-info/SOURCES.txt'
running build_ext
Creating /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/kinto-client.egg-link (link to .)
Adding kinto-client 1.0.1.dev0 to easy-install.pth file
Installed /home/mathieu/Code/Mozilla/kinto.py
Processing dependencies for kinto-client==1.0.1.dev0
Searching for six
Reading https://pypi.python.org/simple/six/
Best match: six 1.10.0
Downloading https://pypi.python.org/packages/source/s/six/six-1.10.0.tar.gz#md5=34eed507548117b2ab523ab14b2f8b55
Processing six-1.10.0.tar.gz
Writing /tmp/easy_install-D_w1p4/six-1.10.0/setup.cfg
Running six-1.10.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-D_w1p4/six-1.10.0/egg-dist-tmp-imhqRW
no previously-included directories found matching 'documentation/_build'
zip_safe flag not set; analyzing archive contents...
six: module references __path__
creating /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/six-1.10.0-py2.7.egg
Extracting six-1.10.0-py2.7.egg to /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages
Adding six 1.10.0 to easy-install.pth file
Installed /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/six-1.10.0-py2.7.egg
Searching for unidecode
Reading https://pypi.python.org/simple/unidecode/
Best match: Unidecode 0.4.18
Downloading https://pypi.python.org/packages/source/U/Unidecode/Unidecode-0.04.18.tar.gz#md5=148a6b31e4fdedc444fe2c6430af3f82
Processing Unidecode-0.04.18.tar.gz
Writing /tmp/easy_install-MVJdpM/Unidecode-0.04.18/setup.cfg
Running Unidecode-0.04.18/setup.py -q bdist_egg --dist-dir /tmp/easy_install-MVJdpM/Unidecode-0.04.18/egg-dist-tmp-02wjeh
/home/mathieu/Code/Mozilla/kinto.py/.venv/local/lib/python2.7/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '0.04.18' to '0.4.18'
normalized_version,
zip_safe flag not set; analyzing archive contents...
Moving Unidecode-0.4.18-py2.7.egg to /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages
Adding Unidecode 0.4.18 to easy-install.pth file
Installing unidecode script to /home/mathieu/Code/Mozilla/kinto.py/.venv/bin
Installed /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/Unidecode-0.4.18-py2.7.egg
Searching for requests-hawk
Reading https://pypi.python.org/simple/requests-hawk/
Best match: requests-hawk 0.2.1
Downloading https://pypi.python.org/packages/source/r/requests-hawk/requests-hawk-0.2.1.tar.gz#md5=104fbd8630e724ddb15a99886caf0ac7
Processing requests-hawk-0.2.1.tar.gz
Writing /tmp/easy_install-afAjSJ/requests-hawk-0.2.1/setup.cfg
Running requests-hawk-0.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-afAjSJ/requests-hawk-0.2.1/egg-dist-tmp-guXxme
creating /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/requests_hawk-0.2.1-py2.7.egg
Extracting requests_hawk-0.2.1-py2.7.egg to /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages
Adding requests-hawk 0.2.1 to easy-install.pth file
Installed /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/requests_hawk-0.2.1-py2.7.egg
Searching for requests
Best match: requests hawk-0.2.1
Downloading https://pypi.python.org/packages/source/r/requests-hawk/requests-hawk-0.2.1.tar.gz#md5=104fbd8630e724ddb15a99886caf0ac7
Processing requests-hawk-0.2.1.tar.gz
Writing /tmp/easy_install-Z2rl_g/requests-hawk-0.2.1/setup.cfg
Running requests-hawk-0.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Z2rl_g/requests-hawk-0.2.1/egg-dist-tmp-g0jI9X
removing '/home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/requests_hawk-0.2.1-py2.7.egg' (and everything under it)
creating /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/requests_hawk-0.2.1-py2.7.egg
Extracting requests_hawk-0.2.1-py2.7.egg to /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages
requests-hawk 0.2.1 is already the active version in easy-install.pth
Installed /home/mathieu/Code/Mozilla/kinto.py/.venv/lib/python2.7/site-packages/requests_hawk-0.2.1-py2.7.egg
error: The 'requests' distribution was not found and is required by kinto-client
Makefile:15: recipe for target '.venv/.install.stamp' failed
make: *** [.venv/.install.stamp] Error 1
Re-run and error was gone.
While creating a collection and you don't have the permission to do so, you might endup having:
File "kinto_http/session.py", line 84, in request message = '{0} - {1}'.format(resp.status_code, resp.json())
collection_timestamp() method ?
Currently the list_records method does not provide the headers obtained.
The (only?) way of doing it presently is:
records = client.list_records(_since=0, _sort="-last_modified")
timestamp = '%s' % records[0]['last_modified']While debugging something else, I found out that a request forged by kinto-http contained an extra key data.
request.body (on the server side):
{
"requests": [{
"body": {
"data": {
"driverVersionComparator": "LESS_THAN_OR_EQUAL",
"driverVersion": "6.1.7601.18328",
"vendor": "0x1002",
"blockID": "g1239",
"enabled": true,
"feature": "DIRECT3D_10_LAYERS",
"devices": ["0x4396"],
"id": "cf217981-2c42-76bf-df19-4324a194a075",
"details": {
"why": "test",
"who": "test",
"name": "GFX1-Val",
"bug": "https://bugzilla.mozilla.org/show_bug.cgi?id=838845",
"created": "2016-10-18T09:29:16Z"
},
"os": "All",
"featureStatus": "BLOCKED_DRIVER_VERSION"
}
},
"path": "/buckets/staging/collections/gfx/records/cf217981-2c42-76bf-df19-4324a194a075",
"method": "PATCH"
}, {...}, {...}],
"data": {}
}@leplatrem : When we do a DELETE HTTP request on a single group that does not exist, the server returns a 404 (which is interpreted as an error). However, when we do a DELETE on a plural endpoint (like here /buckets/x/groups), the server returns an empty list if no record were deleted.
So, if_not_exists argument is not needed for plural endpoints.
Use case:
#0. initialize source bucket/collection (if necessary)
client.create_bucket(if_not_exists=True)
try:
client.delete_collection()
except kinto_exceptions.KintoException:
pass
client.create_collection(if_not_exists=True)Would become:
#0. initialize source bucket/collection (if necessary)
client.create_bucket(if_not_exists=True)
client.delete_collection(if_exists=True)
client.create_collection(if_not_exists=True)c = kinto_http.Client("https://kinto.dev.mozaws.net/v1", auth=("user", "pass"), bucket="test", collection="412")
client.session.nb_retry = 1
records = c.get_records()
r = records[0]
c.update_record(r)Fails with 412 !!!
On KintoException after the batch, we need to have more informations
kinto_http.exceptions.KintoException: (KintoException(...), "403 - {'errno': 121, 'error': 'Forbidden', 'code': 403, 'message': 'This user cannot access this resource.'}")
get_buckets()patch_bucket()delete_buckets()delete_collections(bucket=None)delete_records(bucket=None, collection=None)Default would be error for example, but adding debug output would help a lot sometimes
Current:
parser = cli_utils.add_parser_options(...)
args = parser.parse_args(sys.argv[1:])
client = cli_utils.create_client_from_args(args)
client.session.nb_retry = 3Suggestion #1:
parser = cli_utils.add_parser_options(...)
args = parser.parse_args(sys.argv[1:])
client = cli_utils.create_client_from_args(args, retry=3)Suggestion #2:
With new CLI option like python main.py --retry=3
parser = cli_utils.add_parser_options(..., default_retry=3)
args = parser.parse_args(sys.argv[1:])
client = cli_utils.create_client_from_args(args)Some tests would benefit from a refactor. I'm mostly thinking about the logic contained in the _get_or_create private method, which is not directly tested but via its callers.
One solution here would be to create a base class with the behaviours we want to test and then derive multiple classes with the different inputs (bucket, collection and record).
The if_not_exists behavior doesn't ignore the 412 if they happen in a batch.
Update collection fails with omitted collection argument.
client.update_collection(collection_id, bucket=bucket_id, data=collection)Raises:
kinto_http.exceptions.KintoException: (KintoException(...), 'Cannot get collection endpoint, collection is missing')
Instead you have to do
client.update_collection(collection=collection_id, bucket=bucket_id, data=collection)I guess we should fix that to make collections compatible with other resources.
Traceback (most recent call last):
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "scrape_archives.py", line 29, in publish_records
batch.create_record(record, safe=False)
File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
next(self.gen)
File "/home/mathieu/Code/Mozilla/buildhub/jobs/.venv3/lib/python3.5/site-packages/kinto_http/__init__.py", line 101, in batch
batch_session.send()
File "/home/mathieu/Code/Mozilla/buildhub/jobs/.venv3/lib/python3.5/site-packages/kinto_http/batch.py", line 52, in send
resp, headers = self.session.request(**kwargs)
File "/home/mathieu/Code/Mozilla/buildhub/jobs/.venv3/lib/python3.5/site-packages/kinto_http/session.py", line 80, in request
time.sleep(retry_after)
TypeError: an integer is required (got type str)
client = source.clone(auth=("reviewer", ""))
AttributeError: You cannot specify session and server_url or auth. Chose either session or (auth + server_url).
./kinto_client/__init__.py:
154 : print("paginated")
We should release a 2.0.1 with the fix.
See https://github.com/Kinto/kinto-http.js/#listing-bucket-history
They currently take the id of the collection / record as the first argument, whereas the data is the second one.
I've found myself in a bunch of situations where I was passing a dict as the first argument. We should either consider changing the signature to have the data as the first argument or issue a warning if the id is a python dict.
Line 45 in 8fc639e
https://pypi.python.org/pypi/kinto-http/
That suggests kinto-http works with Python 2 which it doesn't for 8.0+.
This issue covers updating the trove classifiers and pushing a new release which will update the meta information at the bottom of the PyPI page.
I want to provide custom JSON-merge headers on a call to client.patch_record() but apparently I can't provide them.
It would better to accept custom headers like on the JS client, that would be merged with the ones the session (like authorization)
Ideally it should just be «requests» :) PyFxa and request-hawk pull too much stuff and are probably out of scope
See usage in test_client.py
We should raise an exception when create_collection returns a 403, asking the user "Are you sure the bucket exists?".
We need to think of a way to sync a kinto collection locally using kinto.py
Provided a public key, the client could verify signatures during get_records() calls...
Currently it only contains python-requests/<version>
The thing is to make sure to create all the record from the lower last_modified record to the greatest one so that we keep the exact same last_modified value for the records.
The syntax could be:
kinto move-collection-records /buckets/fennec/collections/experiments/records /buckets/fennec-staging/collections/experiments/records
Currently it only supports write operations, so it is not possible to do the retrieval of information inside a batch.
with client.batch() as batch:
batch.get_record(...)
batch.get_record(...)
records = batch.results() batch.create_record(data=record)
File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
next(self.gen)
File "/home/mathieu/Code/Mozilla/kinto.py/kinto_http/__init__.py", line 108, in batch
batch_session.send()
File "/home/mathieu/Code/Mozilla/kinto.py/kinto_http/batch.py", line 81, in send
raise exception
kinto_http.exceptions.KintoException: <unprintable KintoException object>
Ref #147
Today we handle bucket and collection creation only if the remote_client is created.
The bucket creation and collection creation may be done even if the remote_client is already created or passed along.
Detect their presence and present a clear message to users
From client:
>>> client.get_bucket(bucket="source")
Traceback (most recent call last):
File "output-yaml.py", line 43, in <module>
main()
File "output-yaml.py", line 38, in main
result = introspect_server(client)
File "output-yaml.py", line 16, in introspect_server
for bucket in buckets
File "output-yaml.py", line 16, in <dictcomp>
for bucket in buckets
File "output-yaml.py", line 22, in introspect_bucket
print client.get_bucket(bid)
File "/home/mathieu/Code/Mozilla/kinto.py/kinto_http/__init__.py", line 243, in get_bucket
raise BucketNotFound(bucket or self._bucket_name, e)
kinto_http.exceptions.BucketNotFound: (BucketNotFound(...), u'source')
From server:
"GET /v1/buckets/source" 401 (3 ms) request.summary agent=python-requests/2.9.1 authn_type=None errno=104 lang=None time=2016-10-28T12:58:09 uid=None
Currently the error is raised once every batch is executed.
I am importing 10000 records, it represents 400 POST /batch requests. If the first one fails (e.g. wrong id format in some record), I don't want the next 399 requests to be executed.
In other words, the status of the responses should occur in the batch object instead of the client.
Thoughts?
Currently there is no way to only return one page. The client follows Next-Page until it's gone through everything.
I suggest we do like kinto-http.js with a pages argument, defaulted to 1 if limit is specified.
Note that when limit is not specified in arguments (but defined on the server for example), we would follow every pages (pages = float('inf'))
When a 503 is received (high load, throttling), I loose the context of my batch and it is very hard to resume.
For example here records comes from here and contains 10000 entries.
with client.batch(bucket='guessign', collection='it') as batch:
for record in records:
batch.create_record(id=str(uuid.uuid4()), data=record)
I am thinking of a retry option, that would read the retry-after header and try again.
Check its presence and present a clear message to user.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.